Chimps have crazy grip/arm strength, humans might have more powerful legs but you cant really use your legs to tear flesh, break fingers and take out someones eyeballs, legs can be used for blunt force trauma but are less reliable in a close quarter all out melee fight with a chimpanzee. Chimps are fast, not human kinda fast but more like extreme quickness/explosiveness, like when you see dogs go apeshit and jump and bite a mans face before he can even react.
In a close quarter fight with chimp, the chimp would use his grip/hand strength you rip off your flesh, balls, eyes, fingers and it would do so in surprising speed and efficiency. I also think you are underestimating their level of co-ordination, they may have less fine control of their muscles than a human but a chimp can pick a branch several feet in the air and grab the exact spot he intended and pull himself.
Humans have used our intelligence to make and use tools for 1000s of years, we haven't been swinging from trees, getting into vicious hand to hand fights on a daily basis, living in jungles, etc. Physicality is what they do, mental prowess is what we do.
Chimps are extremely dangerous animals, and can cause an extreme amount of damage to any human in a short amount of time. IMO they have the advantage in a physical altercation.