I just finished a two-day shotgun project last week. 7 slugs & 4 buck loads through two 18-inch 870s. First- if you haven't already, get either a Limb Saver or a KICK-EEZE pad installed on your gun. Then add either a SideSaddle six-round ammo carrier or a buttstock-mounted six-round carrier. You can call it tactical if you want to feel cool about it, but the greater advantage is in adding weight. Another suggestion would be to put a recoil-reducing device inside the buttstock, a Dead Mule or a mercury reducer. They add more weight, and adding weight reduces the slam against your shoulder. These are the cheaper ways to go. A more expensive solution is to get your barrel ported, it does work. The best way is to do most of the above, and send the barrel to Vang Comp for porting, backboring, and forcing cone lengthening. If you've got the money. These are all ways to reduce the pain.
Anyway- a magnum buckshot shell frequently does contain 12 pellets as opposed to the normal 9. All the magnum 12-pellet loads I've tried over the years have kicked like hell and spread rapidly. You get a shorter practical range with more wear on the shoulder and longer recovery time between shots, if that means anything to you. Not worth it to me. The Ranger Magnum 2 3/4-inch 00 buck load Winchester sent has 9 pellets, it surprised me. Good tight pattern, not too hard on the shoulder (in my comped 870, anyway). They also sent their Ranger 9-pellet reduced recoil 00 buck load, that one did better in the standard bore 870 at 7 & 15 yards than the magnum load, but both were roughly the same at 15 inches at 25 yards. In the comped barrel, the magnum load was markedly better at 7 & 15 yards, but the reduced recoil load beat the magnum at 25, and was still inside a silhouette with an 11-inch group at 30 yards.
Shotguns will vary a lot, each one is it's own personality, and two otherwise identical shotguns will probably perform differently from each other. Ammo brands are also highly variable, and it pays to do what you've been doing- try several to find what works in your own gun, if you can.
A general rule of thumb is that higher velocity equals greater buckshot spread. That's not always the case, but I've avoided magnum 00 buck loads because it's held true for me.
Those two Ranger loads were for law enforcement, you may not be able to get any unless you're in the biz. If not, stay with standard vel 2 3/4-inch 00 buck, or drop down to reduced recoil loads if you have to, and go with whatever patterns the best in your shotgun. Also remember that 25 yards will be pushing it for defensive use against people for most standard vel stuff, the majority of the pellets will probably still strike a torso at that distance, but you may have several miss & that decreases effectiveness while it increases liability & risk to bystanders. Reduced recoil loads can take you out to 30, and possibly 35 yards, with the right gun and ammo combination.
Can't help with your lead removal, it may be the ammunition. I have to scrub pretty good through my bores, but nothing like that.
For slugs, consider the newer Remington BuckHammer, any of the Brenneke slug loads, and the Brenneke KO sabot shell that throws a 1-ounce cookie-cutter slug that never touches the barrel walls. Out of my standard cylinder bore barrel I got a three-shot best group at 50 yards of 2 3/4 inches.
Good luck.
Denis