The rich are a very small group of people. Those poor people you don't care that leave involve major players (i.e. teachers, line cooks, custodians, park attendants, etc.). I would argue those poor people leaving have more value than the rich who are supposedly leaving. Your second paragraph is not as valid as it used to be considering the people that have a lot of the money are the same people that are asking people to show up to work in physical locations. People seem to forget that moving does not necessarily solve the problem. Look at what happened when people left California to move to Austin. It was not the boom a lot of rich people thought it would be because you cannot replicate all the same things in different areas.
As far as your real estate comment everyone talks about rich people investing and that is used all the time but if you need to provide for the people that voted you in office why would you ever believe the "free market" will provide adequate housing? I'm not saying the government does not have faults because it absolutely does but the "free market" cares more about ROI than providing affordable housing for the teacher with 2 children.