Is this someone who you work with or is otherwise difficult to avoid? Context? How did this escalate?
I had some encounters like that when I was younger before I started training wrestling, bjj, and boxing. After that, my confidence increased substantially and I put on a lot of weight and strength.
I had an older coworker make a comment about "if I were to kick your ass.." to me and it actually didn't really bother me. He's old, big, and fat with no fighting experience other than tussling with prisoners when he used to be a guard. I feel confident I could take him even though I'm healing from an injury. We have a good coworker relationship anyway now.
I avoid fights unless absolutely necessary and I carry pepper spray in case of something like that. Why risk an injury or legal trouble? Better to be the bigger person and walk away unless that doesn't work and they keeps starting stuff. Even then, fighting is the last option unless they are going wild and need to be stopped. I would just call the cops though unless someone is getting hurt before they can arrive.