We have lots of 40+ guys in our judo club who train and compete. Most of the pure judo guys started as kids and then took 20+ year breaks before coming back as adults, but there are still a fair amount of guys who started at age 35 and older. I'm almost 39 and pretty horrible at judo compared to the "judo guys," but its still fun and at worst I'm still probably better at takedowns than many of my BJJ training partners. If you find a judo club that is ground oriented you'll also learn a lot of different attacks to and from turtle and some other positions.
Judo is also pretty scalable-- I have a fused ankle and if I do a lot of standup sparring I won't be able to walk well for a couple of days afterward. So even though I do judo a lot of the time I just skip randori (standup sparring). It's no big deal, I still get in the warmups, drilling techniques, uchikomis, newaza, etc. I'll probably never get as good as a guy who is able to spar more than me, but I'll be a lot better than guys who aren't doing any judo at all. And it's fun. Just tell the judo coach about your concerns.