Actually, it was the Hauge convention, not the Geneva convention that outlawed expanding bullets. The United States was not a signatory in the Hauge convention and even if we were, it only really applies to war between formal armies in a formally declared war. Regardless, we have chosen to stay with FMJ ammo out of deference to international politics. The United States military has never, to my knowledge, made extensive use of expanding bullets.
As to the reasons for switching to 5.56x45, well there are a variety of reasons dating back to the 1950's. The 30-06 was originally replaced by the 7.62x51 so that the individual soldier could carry more ammunition without increasing the size and weight of the load. This was a concern after our experiences with Chinese human-wave attacks in Korea. We adopted 7.62x51 in the M14 rifle and decided that adding a selector switch to allow for suppressive fire would also be a good idea.
Unfortunately, the M14 is not all that controllable in full-auto due to the fact that it fires a full-power .30 caliber battle-rifle type cartridge. This is a lesson that had been learned by several other militaries who had tried to adopt full-auto battle rifles such as the FG-42. You see, in order for a battle rifle to be controllable in full-auto, it usually has to be pretty large and heavy like the HK G3 or Sig 510/Stg57. Both the Germans and the Russians learned much earlier than we did that by using an intermediate cartridge, you can have more controllable full-auto fire, carry more ammo for the same size and weight load, and still retain effective ballistics out to normal combat ranges (300-400yd).
The original AR-15 rifles that were used by military advisors in Vietnam actually got pretty good reports because the 5.56/.223 cartridge was plenty effective at the ranges typically encountered in Vietnam. Also, the early versions of the rifle used a 1:14 twist rate in their barrels which just barely stabilized a 55grn bullet. This meant that the bullets would fairly routinely de-stabilize and tumble when they hit their targets causing some pretty gruesome wounds similar to that of the old .303 British Mk VII FMJ. However, in order to both improve long-range accuracy and make use of heavier 62, 69, 75, and 77grn bullets, the twist rate of the rifling has been changed to faster and faster rates such as 1:12, 1:9, 1:8, and 1:7 over the years. These faster twist rates better stabilize the bullets and thus reduce their propensity to yaw.
Personally, I don't think that the 5.56x45 cartridge itself is a problem, I think that the way it is often employed is more the issue. To my mind, the 5.56x45 cartridge employs much the same niche as the submachinegun used to: the ability to lay down suppressing fire at short-to-moderate ranges.
I've actually heard and read very few complaints about the effectiveness of the 5.56 cartridge from the Vietnam-era up to the 1980's, most of the complaints from that time period seem to center around the reliability of the M16 rifle (which is a separate issue in and of itself). Instead, most of the complaints that I hear about the 5.56 cartridge seem to be from the last 10-20 years and usually have to do with the cartridge being used at extended ranges such as those encountered in Afghanistan. The better solution than a wholesale change in caliber, I think, is to simply issue scoped M14's more liberally as designated marksman rifles (similar to the way in which the Russians use the SVD Dragunov). The 5.56 cartridge actually works very well when used as originally intended, its when you try to turn it into something that it was never meant to be that you run into problems.
