Consent is not always so simple as a "yes" word. Context should be taken into account. We are not robots, and cannot be expected to sign a document every time we initiate physical contact. There might be a warm moment in a movie between two people who are dating, and the man might feel appropriate to touch the woman. But it could be that he's misjudging the situation and is blown off.
By that point, when you're visibly blown off, it is obvious that you should move no further. That, to me, is the "fine line" between harassing and, well, pushing your luck. But if we judge men as harassers based on all unwanted initial physical contact (no matter how minor), then it's going to poison relations between the two genders.
We are already seeing a counter-phenomenon of women wanting more masculine, daring men, that show initiative. If we take away all initiative from the men, then we doom a lot of men into spending their days in solitude, wondering why they cannot seize any opportunities. To a degree, we already have.