Does sound like social anxiety and yeah it does make sense. People can be a drag. Even friends, sometimes.
Sounds like you might put a bit too much pressure on yourself when preparing to meet friends/people. Like you said, when they're there or you're there, it's fine. So maybe remind yourself more of that in the moments you're feeling nervous about it and don't go over all the things that can or might happen, but just adopt the mindset that you'll handle whatever comes in your way and improvise when the time is there, because you know you can. I know you can.