Abe Lincoln and George Washington were unusual in that they were taller than the average person.
Look at the images of Lincoln and his generals. He almost always towers over them. The average height of the Civil War soldier was 5'8". They were also skinny compared to today's Civil War reenactors. It's one of the reasons why Wicked Spring was filmed in Romania and not here in the U.S. Skinny people. Compare that to the average Walmart Shopper.
OK, I concede there were big people back then too. Ben Butler, Heros von Borcke are examples that big was around too; just not as frequently as today.