Athletes already are a tiny minority of a tiny minority and it only takes one rather sub-par, washed up male athlete, pretending he's a chick to absolutely wreck the best female athlete. Women who can legitimately break a record or snag gold are a true minority. One tranny is all it takes to ruin her chances. The idea that it's not a big deal because it doesn't happen often is extremely short sighted for a few reasons.
I'm an athlete and I certainly give a fuck. People who make claims like the one you are making in your post "vehicle to attack trans -women..." Are the ones who don't actually give a fuck about Athletics and are completely ideologically slanted on the issue.