We have a few hundred thousand meat hunters here in PA that hunt the typical two Saturdays per year that would put the first modest buck they see in the freezer. They must now press on until they get a trophy, or make use of one of the excessive anterless permits the game commission now rationalizes as necessary to control the herd. The runt buck they're passing on don't all grow into trophies.
These meat hunters don't have doe tags? In almost all areas, the buck:doe ratio is AT BEST 1:3, usually more like 1:5. If they're "meat hunters", there's a 75-83% chance they're going to shoot a doe anyway. I hunt some areas where in 5 or so years of hunting I've seen hundreds of deer while on stand or walking in and a grand total of 2 or 3 antlered bucks of ANY size. A "meat hunter" in areas like that certainly couldn't care less about an antler restriction. I'm sure the actual ratio isn't nearly that bad but what difference does it make? What you see is what you can shoot. The sighting ratio is probably 1:75.
Most APRs don't stop them from putting a "modest" buck in their freezer. They stop them from putting BABY bucks in the freezer. I would tend to disagree with the western PA 4 pt per antler restriction as too many bucks will take too long getting there. That should be a temporary measure to build population in an area with too few bucks. Otherwise, no one could argue that a 6 point of a virtually any incarnation isn't "modest".
Plus, even the 4-pt/per restriction doesn't require a deer to be a "trophy". I've seen plenty of 8 points that wouldn't have measured 80 inches if anyone had even bothered to measure.
Even with the 4-pt/per restriction, your argument only has merit for the first 2 or 3 years of the restriction, at most. After those years, there will be just as many (if not more) eligible bucks as there used to be "modest" bucks, except for the very few that will never grow that big. Almost all bucks will eventually grow at least a small 8-pt rack.
I note that NY and PA agree that almost all bucks older than 6 months breed, most bucks of all ages only sire one set of offspring per year and the vast majority of bucks killed are killed AFTER the breeding season. Therefore, whatever deer gets killed has little or no effect on breeding. They've all already ensured that their genes will be around next year.