I would say context is key here. I grew up with many black friends (I am not black). We would call each other the N word. I never use the word anymore, but when I was younger, we thought it was "cool". In fact, most black people I know rarely use the word anymore, either.
Ever called a friend a "mother fucker" in a non-disparaging way? Ever used the same term while applying a negative connotation? I know I have. Context.
It isn't necessarily the word, but how it is used. The word "bad" does not always mean "bad" in the traditional sense.
To quote Run DMC: Not bad meaning bad but bad meaning good
There have been multiple instances of whites killing blacks and vice-versa that are not considered racist crimes.
People just choose to find racism when they want to go looking for it. It's not a double standard, it's cherry picking instances that fit your confirmation bias and using that as your litmus test.