I'm not an ex-hunter, but someone just trying to get into the "sport". Speaking as someone who's young and has little familial link to the local land, and marginal acquaintance with local land owners, the two biggest inhibitors are:
1) Access to land. It seems to me that hunting has, traditionally, been something that's kept in the family: you hunt with your family, you clean guns with your family, et cetera. Your family hunting grounds, whether it's your land or public land you know is good, is your's. You may bring close friends (hunting buddies) with you from time to time, but for the most part, it's family. This leads to new hunters rarely getting introduced to the sport: hunting is, most definitely, a largely exclusionary, hereditary pasttime.
This is further complicated by the fact that most farmers want to see the game animals on their land as "their's", and that (in this area at least) there are a lot of out-of-state hunters who will pay farmers a lot of money to hunt on their land, especially if there are reports of big bucks. I know of one family that doesn't allow anyone to hunt their land except for themselves, and paying out-of-state hunters; their family only harvests does and the occasional smaller buck for meat to keep the large buck population up to maintain their customers and their reputation as people with good hunting land.
I imagine a large cause of it is the mobility of our society; 'professional' people don't tend to stay in one area these day for more than a couple years at a time, and that's really not enough time to get familiarized with the local hunting environment.
2) Cost. I'm young and I don't have the funds or time to make the necessary networking connections to find land to hunt on, and finding productive private land within 100 miles of me is difficult, because there are a lot of people vying for that land - it gets over-hunted within the first couple days of a season, and deer seem to have an intrinsic sense about them and they seem to avoid that land as soon as the season starts anyway (there's one public area I know of and have hunted for 2 years now which has a very large buck on or around it, but he has been quite elusive). Gas prices (especially now) for transit and scouting, food, equipment, gun, ammunition, licenses, and don't forget time: they all add up, and for the 'beginner' hunter are quite inundating (the perception popularized by gear retailers that you need this and that doodad, scent, and piece of clothing to hunt successfully doesn't help). It's quite the investment for those one or two shots you'll take at a live animal, and given the modern mentality of immediate gratification, it runs contrary to the cultural grain (which may be a bigger reason for hunting decline than everything else combined).
I think we'll see a bit of a resurgence in the number of hunters with all the combat vets we've now got in the younger generations due to the mentality which is nurtured in the military and the desire to recapture that feeling (a lot of prep for short contact, shooting things, the addictive nature of combat, etc.). We'll see.