The fact that anybody, anybody at all, is telling you that starting at 24 years old is even remotely too late would be laughable if it weren't such damaging advice. Many men aren't even finished growing by 24. Myself included.
Not only will you become more than proficient for the purpose of self-defense, but you could get very good even by local competition standards. That of course, depends on your dedication and intensity.
I would recommend cross-training in BJJ if your main purpose is self-defense, but only because Judo does have a steeper learning curve. Both are great.
Put it this way. If you trained 3-4 times a week, every week, along with keeping in shape; by this time next year, you'll be a whole different man and wouldn't sweat the kinds of altercations like the one you mentioned.
Yes, do it. It will change your life.