This is actually a really question to answer when you know the REAL history of BJJ. A good book I suggest is Opening the Closed Guard by Robert Drysdale. Drysdale also does a bunch of videos on Youtube about it if you want the abridged version. Which leads to the below...
Why didn't he take up the challenge? Did he know his judo was too outdated on the ground?
BJJ originated from Judo. Up until the late 90s/early 00s (aka post UFC 1), BJJ was almost indistinguishable from Judo on the ground. What the Gracies did for almost 80 years prior to that was just Judo with rules tweaked to grapple a bit more on the ground. But even then, watch any of the early Gracie or BJJ vids, and no one is pulling guard or butt scooting. They're all trying to do takedowns. They're also not doing anything that wouldn't be found in normal competition Judo matches of that era. Below is a video from 1949 of a Judo club in England. The second video shows BJJ. Its undated, but probably post 1950s based on Helio's age in the film. What the Judoka in the first video and Gracies in the second video are practicing look almost identical.
Why is this all relevant? The Gracie's had a long history with Judoka leading up to this between the 1930s - 1950s with mixed to bad results. Some of those matches included the Ono Brothers, Takeo Yano, and Kimura. There were other judoka they faced, but I can't recall their names right now. Long story short, the Gracie's always knew that judoka had a leg up on them, which is why many suspect they started focusing more on the ground later on to negate the advantage on the feet.
Would they ever openly say that back then or even today? Obviously not. But you have to realize that Judo was not only 30+ years older than BJJ (so more institutional knowledge, skill, and competitors), but it had regular tournaments that were held for competitors before it became an Olympic sport in the 1960s. So by the time it became an Olympic sport, you already had plenty of well trained fighters and competitors. Meanwhile, BJJ was just trying to stay afloat. There wasn't anything like the IBJJF back then either. Maybe Vale Tudo and Luta Livre, but the bigger talent pool (by number of competitors and practioners) was still in Judo. So going up against judoka of that era while BJJ was still a new thing would have been the equivalent of putting a seasoned and athletic blackbelt against a fresh, unathletic blue belt. I'm pretty sure back then, Judo was a far more popular sport in Brazil (I believe that's still true today). Primarily because its state run.
When you realize all of this, it makes it obvious too why the Gracies would have probably avoided a judoka like Gene Lebell who had a proven competitive record while they were trying to promote BJJ. It also makes it obvious why Rorion Gracie never invited a Judoka (or Samboist) to compete in UFC 1. They likely would have worn a gi and had just as much of a competitive advantage as Royce did.
