I had an issue like this with my younger brothers. My parents were gigantic pussies aka "pacifists" and I, in turn, revolted against that myself. I told my brothers not to take any bullshit and that it was better to lose a fight than to walk away and feel like a coward. They ended up getting in trouble, of course, and eventually were considered the bullies. I have mixed feelings about that now as an adult but in the end I think you have to pick yours over theirs.
Now my daughter is a pacifist. See how that works out sometimes? She tends to just wait for the trouble to go away but recently she was put in a situation where she had to defend herself and she cracked that bitch right in the mouth. One and done!!! Again, kinda mixed feelings but....I kind of also think it was fucking awesome. For her. Nice to know she will defend herself when it really counts.
also