It's annoying because a lot of the time, the fighter is compelled to quit after being fouled. Primarily because the refs don't penalize quickly for fouls.
Put yourself in a fighter's shoes. You're eye poked in round 1. The fight seems close. Your eye hurts and you're not seeing well, but you can continue. You, being a savvy person, realize that if you say you can't see, the fight will either end in a no contest or a disqualification, at the discretion of the ref, so it's either neutral or a win. If you continue, you are at a handicap for the rest of the fight, because the other fighter fouled you. He will not be penalized, only warned, because it's the first incident of the fight. Now you could lose, and you're fighting at a disadvantage. Why the hell would you continue when you're at a disadvantage, but your opponent isn't? It's not about being a man or a warrior. It's about being smart.
Now, imagine if every foul, whether intentional unintentional resulted in a point deduction. Not only would you see less fighters sticking their hands out or kicking low, you'd also see a lot more fighters willing to continue after being fouled because their opponent is now at a scoring disadvantage, so they have a much better chance to win or draw.