Alternative calibers for the AR are commercially priced - only military surplus is ever cheap, and only in what calibers are currently used.
If anything, pursuing cheap ammo cost by purchasing military ammo - that's all you can do - limits getting the right performance you need to do the job hunting. Even worse, hunting ammo isn't military surplus priced, it's commercial and right back into the .65 to $1 a round class again. That's not a major problem - most hunters don't shoot a box of ammo per season, except on hogs.
Is $1 a round ammo horribly expensive? For some, yes, especially if they plan on shooting a lot more than one box a year. At that point, it makes economic sense to load your own ammo. For some reason there are people who can't see this as cheaper and demand that business violate the rules of profit and sell them premium ammo below cost in pallet quantities. It would be funny until you realize they are actually serious. Go figure.
Basically, it's not possible, period. If ammo costs are a big issue - stick to military surplus, but don't expect to hunt with it unless your states regulations allow using Full Metal Jacket. It's not considered an acceptable practice anyway.
For a hunting AR, the list of alternative calibers is pretty long. For the purposes of hunting - and the range and game still weren't spelled out, two calibers are the most popular and in stock on shelves as barrels and bullets, 6.8SPC, and 6.5Grendel. The first was designed as an improvement over the 5.56, with 30% more speed than x39, and 40% more power, from a 14.5" barrel. The second was designed as a complete opposite, a long range paper target shooter at 600m with a high BC bullet coming out of a 20"+ barrel.
6.8SPC has taken off, with dozens of suppliers offering barrels, uppers, and over 4 dozen loads, with more coming out of research and development still ongoing. For 6.5G, the owner has restrictive agreements on using the technology, look for alternates like the .264LBC for suppliers. In terms of what's on the market, it's a poor second, even though it does exactly what it was designed to do - launch a high BC bullet accurately for long distances. However, there's some disagreement if that's necessary at hunting ranges. Nonetheless, it will work, too.
Check both caliber specific forums for more info. There's currently a thread on 68forums on exactly the subject, and plenty of data. Arf.com also has a alternate caliber forum, lots of lively exchange there, too.
I'm finishing up building an AR in one of the calibers, I found a lot of information in the Build It Yourself forum at arf.com, with some of the best assembly tips out there. Special Tools Are Not Needed, although some help better than others.
The major point to building won't be saving any money, it's getting exactly the parts you want in one rifle. Choose the caliber and appropriate matching barrel length - 18" on 6.8 doesn't have a significant advantage - then what upper supports the sights/optics that will work for the typical distances. After that, the stocks, grip, and handguards. Quad rails aren't necessarily any advantage if the barrel is run of the mill accurate. Last, the trigger, and since hunting guns are at best a 1.5MOA proposition, with some rough handling in the field, a target trigger isn't necessarily the best or even safest choice. An adjustable takeup screw is preferred, just like on the millions of bolt guns sold since the 1950's.
Assembly isn't the rocket science some make it out to be, what counts is getting parts that are compatible working together that keep the bolt cycle in it's optimum and most tolerant window. Another pitfall to avoid is thinking that if the military spec'd it,that's best, when in fact, it's just protection to get the taxpayer's money's worth. Each and every part on the AR15 can be made better, the real question is, "Is it really necessary for MY gun?"
Deleting the Forward Assist on varmint and hunting guns comes to mind. It's just not necessary - but it's also hard to find them for less than a typical A3. The industry is geared to put them out in volume, so they are common and dirt cheap on occasion. I paid $48 for a blem stripped LAR A3 FA upper, a non FA was $125. It pays to shop.
Arf.com has a continuing thread reporting the current cheap parts deals, very helpful finding bargains right there from the hundreds of vendors.
Building an AR is really an exercise in research, but you CAN find it on the net, something never before available to the American shooter. Most of the information was armorer and designer level, now it's a culture of helping others understand the operation and action for what it REALLY is, not the myth and misinformation spread among the non-users. Plenty of makers and assemblers will help and inform right down to the nitty gritty details.