I would say the limiting factor in this case is the iron sights. Most people can master iron sights to a certain point, but getting beyond that point is difficult. Most people find that a scope significantly decreases their group size at 100 yards.
You shoot iron sights better than me. I think the best I can do is about a 3 inch group at 100 yards (beer can / clay bird accuracy) I just can't see the target well enough to get it centered over the front sight post. I can easily shoot a 1" group with a modest 4X scope..
I bet your Colt AR is capable of better than 2" group at 100 yards, assuming you find the right ammo for it.