I think the average person who doesn't compete in sports and has a job they aren't satisfied with and somewhat envies athletes of various stripes and military men and warriors/athletes of yore just wants to think of himself as a badass. That's his definition of fit.
The problem is that he moves so randomly from goal to goal that he never really accomplishes any and just keeps chasing ghosts. He wants to bench 275, then he wants to be able to do 20 pulls up, then he wants to be able to run a 5 minute mile, then he wants to be able to look like Gerard Butler from 300, then he wants to be able to damn near kill people like Lesnar and Carwin, then he wants to be like Bruce Wayne bacause he's seen Batman Begins or read a few comics, then he wants to be like a Navy Seal because he saw some documentary about how difficult their training is, then he wants to deadlift like Bolton or KK because he saw a video on youtube, then he wants to be able to do one armed pullups or one finger pushups like Bruce Lee, then he wants to street fight like Kimbo Slice because he saw a video, then he wants to be able to rush the passer like Demarcus Ware because he saw a game, etc. etc. etc.
Here's a quote from Neal Stephenson in Snow Crash that I think parallels a lot of males' mentality when it comes to their potential and ill-defined and unreachable goals.
Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. If my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, and devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad. Hiro used to feel this way, too, but then he ran into Raven. In a way, this was liberating. He no longer has to worry about being the baddest motherfucker in the world. The position is taken.
The problem is that most people just aren't satisfied setting modest goals like get stronger each week or each month and sticking to them one at a time. They have to aim for some sort of super soldier/renaissance man that is great at EVERYTHING status because they have some residual hope that one day they'll get a call from The President asking them to save the world from killer alien ninja robot zombie pirates or some shit.
Edit: I also had somewhat of a minor revelation about Crossfit and things of its ilk. It is geared toward this nebulous jack of all trades fitness as well. But it would seem to take pride in being able to be better than people at something which isn't their sport. For instance, a Xfitter couldn't beat a weightlifter at OL but he could beat him at running 3 miles. He couldn't beat a powerlifter at squatting, but he could beat him at a burpee run. He couldn't beat a sprinter in the 100 m, but he could beat him in a BB complex. It just seems sort of absurd to take pride in beating people... at things they don't train for and not being able to beat people at anything they train for.