I don't think calling someone a racist is the same as hate speech. IMO, it's no different than calling someone a liar or a thief. It might be offensive if unfounded. But it's also an expression of how someone interprets another's actions or speech.
If my wife says something I find offensive, saying that she's rude or that she's lying or that she's inconsiderate, etc. is me expressing how I perceive her words. If she doesn't want me to call her a liar or rude or whatever, the answer isn't for me to not express myself. The answer is to better understand why I feel that way and then decide if my feelings are justification for a change of behavior on her part.
I understand that no one wants to be called racist or transphobic but it's nowhere near the same as hate speech. Hate speech, legally, targets people based on their part of a group (race, gender, sex, etc.). Calling someone racist or transphobic is responding to something that the person has actually said or done, it's not because they're part of some group, it's because of something that they've actually said or done.