If you have someone apply a proper heel hook very slowly, you can feel the tightness in your knee. In training I will tap (and I expect people to tap) when it's locked on and there's no putting the foot back on the floor and no way to turn, or if I get the sense that the attacker has something to prove and goes for my heel.
Find somebody who is good at them and get them to show you what it feels like. You might have to pay close attention the first few times. For me, a heel hook feels kind of like the slack goes out of the joint and things get tight on the inside of my knee. I think toeholds are easier to sense. My entire foot tenses up and I can feel a twinge in my ankle.
In both cases, what really should let you know when to tap is that you understand the submission well enough to know when you are in trouble and when you aren't. But getting a sense of what the sub feels like (from someone who knows what they are doing, and will stop before things get bad even if you don't notice it) isn't a bad place to start.
BTW, when I saw someone who knows what they are doing, I mean someone who legitimately knows what they are doing, not somebody who watched a few YouTube videos and thinks they know heel hooks.
Oh, and when I slap on a heel hook in training, I often don't actually grab the heel. I'll get everything else (legs, angle, etc.) positionally right and then throw the little heel hook gang sign beside their foot, just to say "I've got it, time to tap". Works well in training.
