The Joker's personality can change radically depending on who's writing the character. The only limits I can remember him having is whatever he does must be, "funny" in his eyes. So for example he doesn't abuse animals because they can't get the joke. And no one is allowed to kill Batman except him. Although he'd probably be perfectly happy for Batman to kill him, as it would prove his theory that anyone can be broken by having, "one bad day".
In Killing Joke, he's trying to break Jim Gordon's spirit complete, to drive him insane with rage and grief, as he claims happened to himself. To that end, I have no doubt he would either rape Barbara himself or tell one of his goons to do so. It would not be about sex. It would be about forcing Gordon to watch his daughter being violated.