I have a lot of holes and I think having particular holes (or alternatively particular strengths) in your game actually helps you be successful rolling all the way to upper mid level.
Basically the strengths that you want to be lopsidedly good are guard and passing. Almost all rolls start with guard or passing and they can take up 80% of the roll time.
You don't always submit from side or mount or back, but that's ok because you reset to guard or passing. You don't escape so much but escapes are hard anyway, and that's what guard retention is for.
This might be controversial, but I don't even think it's wrong. Current sport bjj curriculum are like 15% passing 15% guard 15% escapes 15% side and so on. I think beginners would get better much faster with 75% guard and passing.
You sound a lot like me. I agree with you about the focus on guard passing/guard playing. That is the majority of the BJJ matches right there, and it's what makes the biggest difference.
I'm a black belt, but my escape game is not much different than when I was a blue belt. That is to say that I can escape pretty quickly against low level guys, but against more experienced guys, I am pretty much stuck there.
I don't consider that a hole because I've developed my guard to a point where it doesn't matter much anymore. I can easily roll for an entire month without ever getting my guard passed. I train with a group of other black belts pretty regularly, and they all know me as having a frustrating guard so it holds up fine among my peers. There are definitely black belts out there that can smash through it, but those guys are just on a completely different level so it's not like escapes would work on them either. As long as my guard holds up well against my peers (other middle age black belts that are doing something other than full time training), I don't think I have any need for escapes.
If I start needing to improve my escapes, I will work on them. But it hasn't happened yet so I don't see any purpose. I'd rather improve my guard from the bottom, takedowns from the top, etc. I'd rather improve what gets me into a good position that I get to a lot rather than out of a bad one that I don't even end up in anymore.
I'm not worried about the classic Rorion scenario of "What would you do if you woke up in the middle of the night and a burglar was mounted on you?" I'm not worried because 1) I can already escape white belt/blue belt level mounts quickly as it is so that's probably enough for self-defense and 2) I put an alarm system in my house, try to maintain situational awareness, etc. Basically my self-defense strategy is to hopefully detect said hypothetical burglars BEFORE they manage to mount me completely. That way I could use other strategies to prevent being mounted by some burglar in the first place.
So the fact that my escapes are relatively weak for a black belt is an example of how there are things that I don't do much that I do not consider to be holes.