Transvestites aren't transgenders, and fashion changes so radically that what they wore as men's clothing comes closer, by modern standards, to a woman's dress. Furthermore, if the transgender person isn't yet transgender, and isn't cross-dressing when he/she orders, then even this religious text is irrelevant.
Finally, the freedom to practice religion doesn't supersede secular law. We are not a Christian nation. We are a secular nation composed mostly of Christians (for now, at least). We cannot tolerate a precedent that allows a Muslim to refuse service to a Christian because anyone who isn't a Muslim is an "infidel". We must manage a common space.
The best defense of the baker, here, is my previously forwarded arguments about custom orders. It preserves both the CRA and the freedom of religion. It really is a finely-balanced and highly creative compromise. I'm quite proud of it.