You're distorting the truth.
There are two issues that are only tangentially related. First, Linus is an asshole. Imagine the stereotypical socially awkward asperger programmer who doesn't understand why calling people idiots to their face over small disagreements is socially frowned upon..yeah, thats him. His asshole-ishness is infamous and legendary to anyone that knows anything about him. No one is accusing him of being or saying anything racist, sexist, or homophobic. Other people who help maintain the kernel pulled him aside and told him that being an asshole was hurting the project because it was driving developers away...and not specific types of developers..all developers because he's a dick to everyone. He heard them out and apparently realized that he could do better and decided to take time away. SJWs did not drive him away. In fact, a kinder, friendly Linus will rejoin the project at some point.
The second issue is that many open source projects have toxic communities (in terms of them being full of assholes). And because Linus is a major asshole, it fostered an asshole culture within the Linux community. Realizing that it wasn't enough for only Linus to stop his asshole-ish behavior, the maintainers decided to adopt a new code of conduct. They didn't write the CoC themselves instead opting to adopt a third party CoC that has been used successfully by other open source projects (Kubernetes, Ruby, etc) called the Contributor Covenant. If you actually READ the code of conduct, it basically amounts to "don't be a dick" but the alt-righties got triggered because it also included one bullet point about not being racist, sexist, homophobic, etc (among other things including "body shaming") to encourage inclusivity and decided that it was a SJW takeover. Again, the CoC was not adopted because anyone thought that Linus was some sort of bigot. It was adopted because a lot of the developers are dicks and the CoC discourages people from being a dick.