The extension should simply unscrew off the end of the magazine tube (if you have a barrel clamp you may also have to remove that). Once the extension and barrel are removed, the dimples or remnants thereof, if present, should be plainly visible from the outside of the magazine tube. Run your finger around the inside of the magazine tube where the dimples or their remnants are and see if you can feel any obvious burrs or roughness there. If you do, smooth it out with a dremel tool.
Older Remington 870's used a different setup in which there was a ball detent on the front of the barrel ring which mated up to gear-like teeth in the magazine cap or magazine extension. The purpose of this was to keep the magazine cap/extension from unscrewing under use/recoil. Newer 870's have a plastic retainer cap with gear-like teeth that fits over the end of the magazine spring inside the magazine tube. This retainer cap's teeth mate up with the teeth of the magazine cap to serve the same purpose as the old ball detent setup. The purpose of the dimples is to keep the retainer cap from simply turning with the magazine cap and thusly defeating its whole purpose.
If you wish to change your shotgun back to its standard 4-shot magazine, assuming it is of the newer type, you will need to obtain a shorter standard-length magazine spring, a retainer cap, and a magazine cap. With the dimples removed, the retainer cap will simply turn with the magazine cap but it will still be needed to prevent the end of the spring from binding. I had a magazine extension on my 870 at one time, but decided to go back to standard configuration. I found that tightening the magazine cap with a strap wrench gave me sufficient torque to keep if from unscrewing under use.
If you would prefer to keep the extension and there are no obvious burrs or roughness in the magazine tube, I'd thoroughly clean it to make sure that the follower wasn't simply binding on dirt or some other sort of fouling. If the problem still persists, I would guess that the issue lies in the magazine extension or barrel clamp. Some of the cheaper plastic extensions like those from ATI have enough flex in them that they can be distorted to the point of impaired feeding if the barrel clamp is too tight. If you want to use a magazine extension, I'd suggest a high-quality metal one like those available from Choate Machine and Tool.