They could, but this would mean that they would have to deal with decreased productivity. If the upper/middle class ethnics Swedes wish to retain their living standards, then they must construct an "under-class" of foreign-born citizens who will provide cheap labour. The only ones who potentially suffer from this are, of course, the lower class Swedes or previous immigrants, who will be forced to live with the consequences. Not the upper/middle class citizens who live far away from the reality of these immigrant collectives, and will continue to for a long time (but not forever).
A man does not really give a shit if there are cars being burnt around the land, as long as it is not his own car that is being burnt. He can always look at the sweet results of the immigrant scam while washing the windows of his new Ferrari, safely locked behind walls and surveillance cameras, while preaching to others about solidarity.