Well, I did it
I put this off because it sounded difficult. In fact, I tried a different thumb safety that I knew worked in another pistol, but had the same problem.
So I got out my 7/64 drill bit and it went in a little way, then became hard to turn by hand. So I chucked it into my Dremel, which I left unplugged. Then I turned it a couple of times and tried the punger tube again, which then entered the tube without any resistance.
I reassembled and the problem was gone. Safety works smooth as can be.
Here is a pic of the pistol. No smart remarks.