I don't think that's such a bad thing. I listened to GSP's podcast awhile back on JRE and he talks about training with low level sparring partners and how it helps him.
To sum it up, he basically mentioned that training with higher level sparring partners will make you more hesitant in a fight since they didn't give you as many openings, making the fighter being afraid of being caught in the actual fight.
Mentioning that many high level BJJ practitioners would roll with blue belts to boost their confidence and open up their creativity when they're facing higher level competition.
Now if he's ONLY training with guys of this lower caliber it might open up defensive holes too. Keep a balance between training between higher caliber fighters and mid caliber fighter.