You've got to carry what you are comfortable with. Of course, there are reasons why some of us feel the way we do about manual safeties. It's at least worth considering as your comfort level evolves.
I have two problems with manual safeties on carry guns. First, it's complicated. Look at posts 10, 14, and 15. The fact that there was a misunderstanding at all isn't uncommon. There are different kinds of manual safeties. People have different ways of treating them. There isn't a clear standard such as we have for "trigger safety".
That said, the trigger is the one necessary control on modern firearms. You always have to use that one thing. You can tailor your interface with that one thing via a heavier trigger, double action, SA/DA, etc. Once you add a manual safety switch though, you've got two things. That's double the amount of things between you and shots on target in a bad situation that could develop quickly and chaotically.
Think of it this way. You might have someone or something on top of you. You might be pressed against a surface. You might be tangled up. You might be injured. Your thumb might be broken or your hand might be slick. It might not be ideal to need twice the fingers or twice the time to operate two things.
Those are some serious physical possibilities. Getting back to complications, someone with the second thing on their gun needs to remember to operate that second thing regardless of shock, heart rate, adrenaline, etc. I know everyone here trains enough that they'd never have this problem but every once in a while, somebody in the real world does.
Sure, you could just leave it off. Is there any possibility that it accidentally gets turned on? I don't just mean during normal life. How about under those rough possibilities mentioned above?