The reason this is happening, TS, is because your mind and body already KNOW that you aren't able to do much about it (i.e. your fighting abilities are terrible at the moment) one way or another, regardless of what you would attempt.
Your body + nervous system is NOT equipped or able to diffuse a confrontational situation, and even if it WAS, you still wouldn't know/have the tactical angles, timing, movement, etc. ready or available to execute any maneuvers whether planned beforehand or in the moment. Your gut KNOWS you are screwed either way. It is sadly correct.
So... time to change this.
1. Get strong so that people won't want to attack you (it won't be worth it for them, too messy, too much to lose as ALL violent idiots/cowards always have a running Risk:Benefit analysis in the back of their minds before acting). This is the first and easiest step. You'll notice a difference right away in the way you are treated and feel around others. A nice side-effect is that your nerves will be calmer and stronger, having grown several hundred percent (effectively, relative to their current feeble state) just from processing the strain of moderate to grueling exercise. You will also look handsomer, so might as well.
2. Learn how to fight. Be wise with your decision. Pick one of the renowned effective arts/teachers. If you dick around and kid yourself by choosing an ineffective art/teacher, you are setting yourself up for ruin, plus you'll look like (be) a dick while doing it the whole time. Don't be tempted by flashiness. Winning is the ultimate flash, everything else sucks when it goes down.
Wrestling, boxing, kickboxing, judo, MMA. Stuff like that which everyone knows works. Choose one and have fun with it. You don't even have to compete, just get good at sparring, eventually (and especially) with guys in your classes who do compete.
3. When you are able to fight with good technique and are not afraid to get hit in sparring, you will be able to see things from a more realistic, much more relaxed viewpoint. You will NOT lock up and you will be able to out-think/outmaneuver other people in these types of situations, which nearly guarantees victory on your part when you do it smartly (hopefully all the time).
This doesn't have to take years. It should start working in a month. If you do it right and smartly, it doesn't have to suck either. It's not that hard. It'll be the silliest, funnest, most joyful thing you've ever done.
Now you know what to do. Now you have no excuse
Good luck,
Reyes