Basically, if you look at the post-2007 Shooto and Pancrase Rookie and Neo-Blood (respectively) tournament champions, a lot of them retired shortly after winning the tournaments and, between 2007 and 2013, only, like, 15% of the winners and runner-ups from this time are still fighting or doing well. It might even be less.
What happened in 2007 to make that happen, though? Well, there was one big thing: Pride and HERO's-- the two preeminent organizations in Japan for high-level and well-paying competition-- both went under, then Sengoku and DREAM appeared, which failed to capture the previous organizations esteem, and they gradually went under.
The Japanese economy also hit some real hard times during these years, which made it harder for anyone else to wanna take a risk on combat sports, which have always been a volatile field.
Without a high-level way for fighters to compete that doesn't involve traveling halfway across the world, it's understandable that a lot of them would lose interest in the sport. Whether that meant just retiring suddenly or losing a fight and then struggling to regain that eye of the tiger that leads to consistent improvement in skills and producing victories, the end result's the same.
It's kinda like how the scene was in America pre-2005; there were a lotta good fighters, but a lot of promising guys appeared and disappeared because the UFC wasn't making much money, so being a champion there wasn't much motivation, the smaller shows weren't really enough to sustain you, and traveling to Japan wasn't really feasible.
This kind of a downturn affected the entire Japanese scene, but the lightweight division's not exempt from this.
Remember when Cage Force was a thing? The Japanese promotion under the Greatest Common Multiple banner that was one of the few Japanese organizations to hold their fights in a cage, and they had tournaments where the winner got a contract with the UFC? Takeya Mizugaki won the tournament and signed to the WEC off of it (and, since he was ranked, like, 5th or 6th in the world, he got a title shot against Miguel Torres when Bowles was injured), as did Yoshiyuki Yoshida to the UFC.
You know how there's, like, nothing like that now? That's just an example of how different the scene is.
A lot of the fighters that appeared during this time were great-- it's not fair to the fighters who showed up in 2008 to say they just weren't intrinsically as good as the fighters three years ago were-- but circumstances led to them not being able to show the world their worth.
Another thing was that there was a lack of import of international fighters on the local scenes at this point. In Japan, beating foreign opponents is culturally considered to be a highpoint in the martial arts. Americans can beat other Americans or they can beat foreigners and it's fine either way-- Americans don't go like, "Oh shit, this guy just beat a Brazilian!!! That's a fucking accomplishment!"
But, when the Japanese can beat a foreign opponent, it's a big accomplishment. On the smaller scenes in Japan from the '90's to '07, you could still get a handful of foreign fighters appearing in the various organizations, so that meant the Japanese had a lot more opportunities to beat foreign opponents even if they weren't signed to Pride or HERO's, and, when they're able to win those fights, there's a tendency for their self-worth and confidence to skyrocket, which improves performance and, therefore, the gains made from training and the ability to win fights.
It's kinda like sports psychology at that point. That kind of atmosphere in Japan offered the fighters a lot of opportunities to allow themselves to improve their self-confidence and self-esteem to a level that validates their ideas that they could be one of the best in the world, and that kind of mentality goes hand-in-hand with improved performance.
A Japanese fighter beating a lot of foreign opponents is kind of like an American or Brazilian winning the Tachi Palace or King of the Cage or Jungle Fights championship. When those guys win those titles, their confidence and self-worth has a tendency to improve drastically and they start turning a corner in their training, and a lot of western fans completely understand that. That's what it's like when a Japanese guy beats an American. As great as a Grachan championship is, it doesn't mean the same thing to the Japanese that winning a local title in the west does.
The Shooto, DEEP and Pancrase belts are the obvious exception to this rule, and that's why most of the Japanese guys that've come to the UFC and Bellator and done well have been Pancrase, DEEP or Shooto champions.
There're some other big reasons-- the lack of traveling experience and having to make the horrendous 20-hour east-west flight to America on your first trip is one of 'em, so the Japanese lightweights who were able to make a buzz ended up underperforming in their first UFC fights and then weren't the same by the time they got cut.
The point is that one of the reasons this led to so many great Japanese lightweights is cuz' lightweight is kind of like the bridging point between the two parts of the world. There're a lot of great lightweight-sized Japanese fighters and great lightweight-sized western fighters. So these phenomenon were able to match up and led to a lot of Japanese fighters being ranked.
