If you run a search on this, you're going to find that it is one of those questions that has been asked an answered a million times. The short answer is that if you carry your gun regularly, and especially if you remove the pistol from the holster regularly, you are going to get wear. It is going to happen - accept that now and move forward. It really isn't a big deal and it is a thing that is part and parcel to carrying a firearm.
Leather is going to wear your firearm slower, but to a greater degree - it tends to hug the whole gun, and over time it will dull the finish (nearly) all over.
A kydex (polymer) will probably wear your finish faster, but in isolated spots. Kydex holsters do their best to grip the gun by the trigger guard area (thereby reducing wear on the slide), but you will probably get fast wear towards the muzzle. However, the wear is usually limited to those isolated spots. I prefer this to the "overall, slow" wear of leather.
Nylon is junk - I would write that off altogether. Most nylon holsters are of sub-par quality, re-holstering with one hand is generally not possible with this style of holster, and in my experience, they just plain do not carry well.
I've never tried suede, but I would avoid linings - they tend to accelerate and amplify problems regard junk like dirt that inevitably finds its way into the holster.
Comfort is subjective - I like the small footprint of kydex - it doesn't bother me at all like a larger leather rig does. A lot of guys like a leather/kydex hybrid - leather facing the body, kydex on the outside. You should probably consider looking into this as well.