They were at a bar, where the Seals gather for a yearly reunion. I would imagine if there was a fight, others would have joined in, which there is no reports. Ventura is also a pretty patriotic veteran, who may bash the leaders, but I find it hard to believe he would be reveling in the death of a solider.
I bet Kyle saw him in the bar, and jealously talked shit about him, with his buddies. I doubt he ever even talked to Ventura, but when writing a book, you gotta bring up the most famous SEAL, even if you don't bring him up by name in the book.