Another part of the problem is that people are used to having diets described to them in ten-words-or-less. You know, "the (insert food here) diet," "the (insert food/allergen/whatever here) free diet," a "(high/low/some measure of quantity) (insert nutrient here) diet," and so forth.
In reality, people who have given serious thought to how they eat can rarely describe their diet and the principles it is based on concisely and simply enough for the layman to understand. And real-life conversations have no FAQs to send people to
In other words, when people ask you about your diet, they want a one-sentence explanation. You don't have a one-sentence explanation to give them, because you're a reasonable person who knows these things aren't easily boiled down to a single sentence, especially when they go against conventional wisdom.
And if it take more than a sentence to describe your diet, you're some kind of neurotic freak. In fact, you probably have an eating disorder. (That's sarcasm, BTW).