Weight classes were created in combat sports to have as much of an even playing field as possible and make them a sport where skills and athletic abilities / conditioning make the difference.
If we take your analogy, when 2 people fight if nobody cares about who's bigger then nobody cares about who's armed with what either or who's on the better performance enhancing drugs. So in that sense of real combat where we don't try to create an even playing field, you'd allow fighters to be as big as they want but also be equipped with what they want and take whatever substances they want. If what you want is real combat, then in real combat the small guy isn't going to play on an even playing field with the big guy, and most likely use better weapons, tools, numbers or environment to his advantage. It would be like racing where there's no restrictions about the vehicle you use, then it would be more about who's got the better vehicle than who's the better driver.
As such we conclude if by "fight" you mean "real combat" then it's never about making an even field in the first place and as such we should not have any rules of restrictions. And that's the opposite of what a sport is.
I am very glad lower weight classes exist because they've helped display higher level of skills in the techniques allowed within the rule-sets of those combat sports. I have more interest in who's more skilled in a combat sport and see how they display it than who's the bigger stronger guy with no technique but just pure size and strength to his advantage.
