Agree with Justin10mm. For the price and overall package, the marlin 795 is matched by none. For the every-man, the Ruger 10/22 is the way to go, but out of the box it's a little less accurate, as the 795 has Marlin's "Micro Groove Rifling". Another advantage of the 10/22 is that customization is endless. You can make it look like anything you want! If you're going to buy a gun and leave it standard, go with the 795.
For me, I bought the 795 because I was low on cash and didn't think I'd want to customize my gun. I guess I was wrong. I'm kinda dissapointed and I'll have to pick up a 10/22 when I get around to it. So like I said, think it out really well before you have to buy both and waste money.
If you want to go down the bolt-action path, which I absolutely love! You'll probably want to go with a Marlin XT-22 or a Savage Mark II. There are others, but these are the best you can get for low-cost ($200-$300 standard). If you'd like to go higher end, I suggest a CZ 452 or the new CZ 453. They're some of the best you can get for their price and they do deliver, I've shot them before.
Something to note is, if you are a good shot, it won't matter. You won't need to spend more money on a CZ, no matter what people say. If this is the case with you, go with the Marlin XT-22. It's basically just as good as the savage and can compete with both the Savage and CZ on both accuracy and looks (to me at least).
As for pistols, there are a number of ways to go (as others said above). You could go cheapy with a Heritage Rough Rider. It's cheap, but it works. Also, look up Legacy Sports International and on their site, go to the Puma tab and click M-1872 .22 Revolver. I heard they're better than the Rough Rider, but I can't be sure. Of course there's always the Ruger line, which I hear nothing but good about. You could go for the Walther P22, but from what I've seen, it's not the most reliable of the bunch. It is small and recoils, so who knows, it may be good practice?
The problem is, there's just so many products to chose from... It's kinda depressing :'(
Anyway, I hope this helped you save some $, as well as give you an idea of some of the options available. Good luck!