I think it's harmless and these babies that cry about toxic workplaces are making a big deal out of nothing.
I used to use the term when dealing with an employee who was being a jerk. I would tell him that we spend a lot of time together, it's like a second family, so he needs to make an effort to not get on everyone's nerves.
But I guess some crybabies don't like their boss using the word "family" because you can't fire your family.
But whatever. It's such a loser attitude to even think about things like that. Those of us that actually work hard don't sit around thinking about getting fired. Like it's not even on my radar.
I suppose I may be laid off one day but if it's gonna happen then it's gonna happen. Using the word "family" won't change that and you'd have to be an idiot to think it would.