I think we're just looking at it from two different perspectives. You're seeing him as not being proven because all he's beaten is Brimage and Holloway, and that point does make sense...but I see him as a top talent because of how he's beaten them and the skills he's shown in beating all of his opponents until now. He's got fast hands, power, very good footwork that sets up opportunities to strike at strange angles, and at the very least his ground game is passable. He's also shown to have heart and adaptability, he hurt himself in his last fight and changed up the gameplan to secure a win using what he's weakest at (grappling). Add that to the fact that he has a tremendous belief in himself and he's a dangerous cat. Cowboy has always been inconsistent, but when he's on he's definitely on, not to mention he's significantly bigger and longer than Conor. I'm not saying he would for sure beat Cowboy or Swanson, but I think his skills have progressed to the point that (despite his competition to this point) he could be a top contender very soon.
I think as a rule you shouldn't believe the hype on fantastic strikers until they fight a decent wrestler, but if I had to guess I would say within a few years Conor's going to fight for the belt.
