You're correct - larger objective lenses are to allow more light into the scope, improving low-light performance. If that's not important to you, it's a good place to save some money since lens cost increases exponentially with diameter.
Have you ever watched a sporting event and seen the huge lenses that the professional photographers use? Those lenses can easily run $3-10k, and it's not because of the magnification (you can get a $10 pocket telescope with higher magnification.) The huge lenses allow more light to enter, so they can use a shorter shutter speed in order to "freeze" the action on the field.