M855 is able to penetrate the U.S. M1 steel helmet at 600m. That was the criteria for development of the cartridge.
That being said, the ability to penetrate the M1 steel helmet at 600m doesn't necessarily make the round suitable for defensive use or medium to large game at that range.
The current 5.56mm military cartridges have their best terminal effects at velocities above 2500 feet per second. At 2500 fps and higher the round breaks at the cannulure and fragments. The rounds fragment most reliablby above 2700 fps, but tests done by Dr. Martin Fackler of the Army's Wound Ballistic Lab placed 2500 fps as the bottom threshold for this effect. These fragments create their own permanent wound channels causing greater damage then the bullet would if it didn't break up.
Yes the bullet tumbles when it hits, but all spitzer type bullets will do this if sufficiently disturbed. It's just basic physics, most of the weight is in the back of the bullet and when it hits something, the weight tends to move in the direction of travel.
M855 is only at optimum velocity for about 125m out of a 14.5 inch M4 carbine, and a little more then 200m out of a 20" M16A2.
So yes the round will penetrate a steel helmet at 600m, but it won't be moving fast enough to fragment and create the massive wounds it does at the higher velocities.
Jeff