How to prevent the hand coming back in the triangle guard?

If it's just the hand, I typically prefer that, since it does nothing to block the choke and makes it almost impossible to defend, your opponent is trapping himself. This is why you won't see triangle choke defenses that involve swimming your hand in. You can block if you get your forearm in, but the hand alone shouldn't work.

 

What kind of lock do you have? That's the first question I would ask, not really your hip height. Your lock should be tight, but then you typically adjust it by rotating. It sounds like you might have the wrong type of leg pressure at different points in your triangle. When this problem happens, I suspect you have a relatively loose lock with your ankles crossed in the middle of your opponent's back.

 

Different people do it differently depending on a variety of factors, including size and flexibility. It would be very hard to describe without being there. Generally with the triangle you are pulling on his elbow while you are clamping with your leg on his neck. That squeezing tension is what isolates his arm and traps him. Probably you are focused on just pulling the guy's head down to break his posture (which is fine) but missing the transition to the "squeeze" because your knees are splaying outward while your feet clamp down on the middle of his upper back.

I actually extend my legs straight during the early phase, using the ankle cross to squeeze really hard. This will not choke him, but it controls his arms and makes it easy to isolate an arm. Then, foot on the hip, I rotate for the lock.

There is a dark art to working your way into the triangle. It's good you are breaking posture, but you will want to combine that with thinking about ways to consistently apply crushing pressure throughout the process.