It depends on the decade. In the 50s, they drank at work. I think though, that in decades before computers, people with office jobs didn’t have bosses that sat on them all day and constantly micromanaged them. So if they finished their work early, they could leave the office and go home, or take a long lunch and meet some friends, or any number of things that didn’t involve them just sitting in their offices.
I remember going to the office with my dad when I was a kid, which was the 80s and early 90s. While they did have computers then, he just stopped by the office to pick up some stuff, then left with me for the day to do whatever it was we were doing (most likely going to an A’s game).
Now, because everything is about cutting costs to maximize revenue, instead of out competing your competition, it makes sense that every manager be accountable for their employees’ whereabouts every second of every day.
