My dad is 66 and until I really noticed him start to slow down a bit physically, I was still shit scared of him as well. The stories about him that I've heard from other people (he'd never talk about his fights without someone else bringing it up) made me weary of really pushing him too far. He was just an untrained tough guy (not toughguy...there is a difference) while I was a wrestler with BJJ, karate, MT, and boxing experience by 21. Still wouldn't have fucked with him then. Might've been able to take him but he would have done some serious damage to me in the process.
Oldman strength is a thing, people.