"Moving heavy things" might just make it worse. More muscle means more water; more water means more bloating. If I recall, the opening poster mentioned once that he weights 210 pounds at 5'7''. Whether that is muscle, fat, or some combination of both, his face will likely have some bloating at that weight for that frame. If he has a very robust frame with very robust jaw and head structure he probably looks fine even with the bloating; but anything less and he will look like shit.
That is the mistake so many men make as well, trying to lift a ton of weights to "look better", when all it usually does is make them look bloated. Meanwhile, the girls that matter - that is, attractive girls in their late teens and early twenties - are really only attracted to men with skinny bodies and skinny faces (with the exception of the occasional man with very gifted and robust facial structure who can get away with having more mass and still look good).
To sum it up in pictures :
Don't be the guys on the right lads. If you have
it, you
have it - and you won't ever have to try; good things just gravitate your way.