Personally I've always made it a habit to NOT release the slide just by thumbing the release down.
I draw back on the slide and depress the release.
I've seen quite a few handguns over the years with hold-open notches rounded off and releases well worn, apparently in large part from years of simply thumbing the release down while the slide is under recoil spring pressure.
That said, my Kahr K9 is the same way. With the magazine in, it's IMPOSSIBLE to work the slide release.
With the magazine out, it's still damned hard.
Strong springs and well-cut notches, I guess.