If you are in a shooting case that goes to trial, anything the prosecutor can do to demonize you WILL be done. If you don't train at all, you'll be painted as negligent. If you train beyond what your state requires, you'll be painted as a Rambo-wannabe.
The main thing is to understand why you are making a specific modification and document it so you can explain it later if need be. For example, if you shoot IDPA, removing the magazine disconnect makes it easier (and safer) to drop your magazine and show clear instead of trying to jam your saluting finger up the magwell to activate the mag disconnect so you can drop the hammer.
Regarding a magazine safety disconnect being removed, if someone is shot accidentally or negligently with that pistol, you've got a problem. All of life involves making risk/benefit determinations, even crossing the street. This is just another one of those areas with a miniscule probability; but serious consequences if it should happen.
If you want to reduce the grittiness without removing the disconnect, polishing the disconnect surface, the magazine surface, or (my favorite) grooving the surface of the mag disconnect to reduce surface area contacting the mag all are helpful - as are high lubricity coatings like chrome or NP3 for those parts.