First off, you're right, I'm probably the last person who would be in a fight because I'll always back away unless I'm forced to defend myself. I'm not some meathead who trains so they can go around and get in fights.
And if the person does have friends, I'm sure they will get involved... at the least they will try and pull you apart, and the most, they will gang up on you and beat you down.
But if you willingly take a person down to the ground, you're asking for trouble. When you're on the ground, you are much more vulnerable to third parties... either by being stomped, kicked, punched - or if you're in the mount, someone can easily come up behind you. When you're standing up, at least you can circle around, or run away. When you're on the ground, you're pretty much committed.