There are two reasons one might want to exercise restraint in modifying a self-defense firearm.
The first, in my opinion, is function.
In my experience, a decent quality firearm from a company with a good reputation is a pretty good bet, in terms of reliable function, especially if you've done a reasonable checkout on it and maintain it properly.
Once that firearm has been tweaked, adjusted, modified, "upgraded" with different parts, etc., things can really change. I realize this is the Handguns section, but back when you could get new Chinese import SKS rifles for around $50 apiece, I used to get a lot of questions about feeding problems with SKS rifles. I would simply ask the person what kind of aftermarket magazine they were using and wait for them to be amazed that I knew, without their saying, that they were using an aftermarket magazine. But it wasn't really that amazing--those guns tended to work and work well the way they come. It was only when the owner started trying to "improve" them that the problems started.
I also have taken Jerry Miculek's advice, provided on Shooting USA, to heart. It was related to competition, not self-defense, but I believe it's still applicable. His comment was that to win, you have to have equipment that works.
I'm also reminded of a case that was outlined some years ago in American Handgunner magazine where a shopkeeper defended himself with a revolver that he had modified with a drop-in spring kit. It wouldn't fire reliably, and although he prevailed, he encountered a number of misfires while shooting his revolver DA, due to the lightened hammer strike.
So the MAIN reason I try to keep my self-defense guns pretty close to the factory configuration has to do with insuring proper function and reliable operation.
The other issue to consider is that one can run afoul of certain aspects of the law if a gun with certain types of modifications is used in self-defense. People like to focus on that, but, in my opinion, it's a secondary concern. You have to live through a self-defense encounter before you deal with the courts, and if your gun doesn't work, your chances of making it that far are reduced significantly.
But legal concerns shouldn't be dismissed--they can cause a tremendous amount of hassle, expense and possibly even loss of freedom.
As far as legal concerns, there's custom and there's custom.
Disabling safeties and going to trigger pulls that depart significantly from the norm for the particular firearm in question are examples of modifications with a potential for legal pitfalls.
Changing grips & sights, adding a "match barrel", altering the handgun to provide a better individual fit without altering function or other similar modifications are all things that would be very unlikely, in my opinion, to generate any sort of issue at all.