I also race bikes, and started shaving this year.
I shaved because coming off on the road and getting rash all up hairy legs sucks.
Because you're a man, you dont have to worry about leaving some patches or whatever, and its a quick rub in the shower every few days, not really a hassle at all.
If you're really working your legs, massaging them back to health with shaved legs is so much nicer, the massager slides a lot better. You're also harder to catch, and you get less matt burn.
Because you're training so much, your legs probably look very strong, so you shouldnt worry about negative comments too much. Its a very different look to when its just done cosmetically on feminine legs. Besides, if comments bother you, you're in shape to rough bullies up anyway, right?