Belts are overrated, but as for hardest.. depends on the gym, the individual, and the coaches. In some ways I think purple is actually harder. There is a lot of talking about how high the quitting rate is around blue belt.
I think the reason for that is because the training wheels come off. Higher belts don't take it easy on you anymore (or much less often). Any Dunning-Kruger effect starts to wear off and the sheer magnitude of things yet to be learned, the hard work or "grind" (ugh I hate that word, can't think of a better one) that will be needed, how competitive and driven the other people who are still doing it are, and the realization that strength, athleticism, matter just as much as they do in every other sport. Makes a lot of people quit because the sheer magnitude is too much