Absolutely size (or at least strength) makes a difference. If it didn't, there wouldn't be weight divisions in all combative sports. By leveling the size playing field, the contest becomes more about skill/preparation than physical advantages.
Fighting, I think, is largely about advantages - exploiting yours, while preventing him from exploiting his. Superior skill is an advantage. Size/Strength is an advantage. Ditto for speed, endurance, numbers, tolerance for pain, a good old fashioned mean streak (vastly underrated, IMHO), weaponry, etc., etc - there are too many to be counted here.
If you can force your advantages on your opponent while preventing him from doing the same to you, you win. If not, you don't.