I ask this question from the viewpoint that soldiers ( all soldiers not just american ) always seem to get a gun that the politicians like instead of what would do the job.
I'm not a Vietnam vet. But I have served in Iraq and Afghanistan. During my years of service I have seen multiple carbine trials looking for the next service rifle. So I'd like to add a few words from my era, specifically about weapons trials and aquisition.
Colt was able to get the M4 adopted without a carbine trial because "the M4 is just a modification to the M16" but then turn around and get a sole provider contract because "the M4 is a novel weapon system" in the patent courts. Colt had been able to have it both ways and their continued focus on .gov contracts has been well documented.
When you take a look at the number of carbines that have all gone through testing; FN SCAR, HK416, XM8, to name a few, we begin to see that the marginal increase in weight reduction does not justify transitioning a 3 million personnel military from the M4/M16 platform to something else. The bean counters will not be denied.
Second, as goes SOCOM the rest of the Military will follow. Want a heavier barrel? The "improved M4 will have the same SOCOM heavy barrel. Want quad rail hand guards? Now available for every M4. PMAGs? Now they have an NSN and can be ordered through normal supply channels (probably the number one reliability enhancer ever in my humble opinion). The reason why is that SOCOM has a better system in place for testing equipment, ie, if they like they buy it and issue it to guys going into harms way. When the guys who went into harms way come back they give a rundown on equipment performance and next thing you know it is SOCOM approved. Big Army is a lot slower than that, even with RFI programs.
During the Vietnam era, aquisition was done through "boards." So the "Infantry Board" was in charge of all the testing, evaluating, and eventual adoption of new weapon systems for the Infantry (and therefore all small arms for the Army). This setup is still here (there is a Test Platoon down at Fort Benning doing nothing but running potential gear through field problem after field problem) but it makes adopting anything a very long process. Rapid Fielding Initiative (RFI) speeds up the process, as well as the Operational Needs Statement (ONS) aquisition process. The problem is that just because you get something on ONS doesn't mean it is going to stick around. All those SAGE EBR M14's? Those are ONS because we don't have the numbers of M110's or M24's to issue to everyone who has an operational requirement for them.
Now SOCOM has been playing around with the 6.8 SPC for a couple years now, but Big Army has been in the process of "improving" the 5.56 with the M855A1 round (my initial impression has been "meh"). So I think that widespread adoption of 6.8 is not going to happen. There is too much infrastructure involved (and too many allies already feeling a budget crunch) to adopt a new round at this time. Adopting a new round will have to wait for peacetime (like the 5.56 being adopted in peacetime, the 7.62x51 adopted in peacetime, the 9mm replacing the 45 in peacetime).
So whether or not the M16/M4 is "the best" will always be open for debate. But definitely so far it has been "good enough" that nothing has been able to dethrown it so far. Now as far as politics influencing Big Army's stance on keeping the M16/M4, I don't have any evidence of that. I can't prove that corruption doesn't exist (no one can, you can't prove a negative) but given the number of Officers, NCO's, and Enlisted personnel who have been involved with carbine tests over the last two decades I really doubt that there is widespread political influence coming down to keep the M16/M4.
Jimro