I usually carry in an OWB holster with a thumb break retaining strap. Occasionally I carry in an ankle rig or a shoulder rig. In all those circumstances, when using the rest room I just leave the gun snapped in the holster and the holster in place.
The only time I ever had particular concern for my safety in a pubic restroom was at the Golden Nugget casino in Las Vegas about 15 years ago. A couple buddies and I were in town for the NASCAR race, and we were staying at the Golden Nugget.
While I was in there another patron was having some kind of melt-down, talking to himself and raving about how he was going to poke somebody's eyes out. Eventually he finished and left. I gave it 5 minutes and then I left too. I never actually saw the guy.