Board member Hummer70, a former two-time national Palma match champion who was also a test director and firearm incident investigator at Aberdeen Proving Grounds, says he has demonstrated that leaving residue in primer pockets increases the rate of throat erosion when it is blown down the bore. This matters most at the highest pressures and bore temperatures, so it isn't much of a problem in the older, lower-pressure handgun cartridges.
As to the effect on accuracy, that probably is invisible in most handgun shooting. In rifle shooting, it is possible the slight cushioning effect of the residue could affect ignition timing and speed, resulting in barrel time variation that could conceivably detune load. That would also allow variability in small mechanical disturbances at the instant the bullet leaves the barrel. These sorts of effects would be most apparent in benchrest competitions where one-hole groups are being produced. They are also most likely only going to show up if the primers are very precisely seated, as most folks get more velocity variation from primer seating irregularity than they would likely see from primer residue cushioning.