I read in some health magazine that men who don't overwork their bodies at young ages will have more testosterone as they hit their 30s and 40s compared to athletic men who play a lot of sports. If you workout in moderation and avoid processed junk you will be able to maintain a youthful strong, healthy body, for years.
I think it has to do with over training causing stress to the body, after intense workouts your testosterone levels drop, another study I read said black men have more testosterone in their youth but during their late 20s and past the age of 30 their test levels drop significantly, while white and Asian men seem to have a peak and rise in test in their late 20s and early 30s.