You say you had a gunsmith check it out . . but just what did he check?
Did he slug the bore and then slug each throat? Rugers are well known for having small throats and throats that don't all match. When they ream the throats, they use a "gang ream" - i.e. three reamers at a time. If one gets worn, it gets changed out. You actually could end up with three adjacent throats each having a different size.
It can be fixed by a good pistol smith who has the equipment - but the bore needs to be slugged as do the throats to see just what you have. You say you get debris back in your fact - that sort of indicates to me that the forcing cone should be checked as well.
You aren't going to get decent accuracy if you have a given bullet size . . . swage it down as it goes through a tight throat and then expect it to swell back up to fit the bore on the way out. And all the expensive sights, etc. aren't going to compensate for that. As an example . . . take a 45 Colt. If you cast a bullet in a Lyman 452 mold and it drops at .454 and you size the bullet to .452 and then you fire it through a throat that is .451 and expect it to go through a .4525 bore . . . and on handguns, there can always be a variance . . that bullet is not going to exit the barrel accurately But, if you cast a .452 from a Lyman mold and it drops at .454 . . . you size it to .453 and you put it through a throat that is reamed to .4535 . . forcing it into a bore of .4525, its going to conform to the actual bore size and exit accurately each time. All things equal, if each throat is size correctly for the bullet diameter and true bore size - each chamber should be consistent (within variables such as + or - .1 or .2 grain weight of powder and grain weight of bullet - if cast) and your groups should tighten up. Even if POI is different than POA . . the groups should tighten.
If you are confident that your gunsmith has actually don a slug of the bore and either slugged the throats or used pin gauges on the throats - then that potential problem should be eliminated. If he did a quick check using a set of dial calipers or some such method of measuring - you can't get a true and accurate measurement.
Good luck to you and I hope you can figure it out so you groups tighten up.