I have trained in Kung Fu, Judo, Brazilian JuJitsu, American Kickboxing, and Thai Boxing. There is no martial art that you can train that will make you a worse fighter (provided you have proper instruction). The way that Tae Kwon Do is taught in the states nowadays is the only thing I would consider an exception to that rule - people are taught to punch from the hip and to do far too complicated kicks that leave their balance (poorly trained) and distance at risk.
Long story short - learn something and become proficient at it and keep it as another tool.
I have competed in Brazilian JuJitsu (Master Relson Gracie) and full contact matches in the past. One of the best moves you can learn is in another style that complements your favorite.
If I had to make a suggestion, I would say Thai Boxing with some knowledge in Brazilian JuJitsu. In a real-world confrontation, one of two things is going to happen:
1) You'll end up on the ground. This is dangerous if your opponent knows how to grapple. With just a few weeks of training you can DOMINATE someone of much larger size on the ground. Since I am relatively small, this is where I take all fights - I can control the speed, intensity, and damage in the fight to my advantage. Perfect case is not beating some drunk idiot within an inch of his life, but holding his face to the floor until authorities show up.
2) You'll stand up and square off. A Thai boxer is going to work to get knees and elbows in, the most effective weapons on the human body for striking. I have been beaten up by Thai boxers and it hurts, a lot.
That being said, I have been thrown on my ass by Judo instructors even when I'm on top of my game. I have been kicked in the face by Kung Fu masters that were quicker and had better technique. It's a matter of finding a style that YOU LIKE and YOU CAN LEARN, PRACTICE, and PERFORM with great skill and YOU ENJOY IT.