Exactly.
The association between test and libido, muscle mass and well being isn't as strong as people think and it's more complicated than a few biomarker. If you feel good and healthy, in your case being able to train hard and have good libido even at 40, I would stay far away from exogenous hormones. I also disagree that you would even need to test for it, as you'd have no idea what the optimal range is for you anyway. No need to create issues that aren't there and risk the possible sides. As you mentioned, even if you don't feel well, lifestyle choices can have greater impact long term.
If you're hypogandic by all means get your medication prescribed.