He could definitely become a contender. He's only 27 and he has a lot of potential still; he's very athletic, very explosive, has a lot of power, and looks like he's trying to improve.
He's only lost against Russow and Overeem, but look at both of those losses: he was dominating Russow bad for the majority of the fight, Russow has knockout power in his hands, though the style he uses doesn't really permit it being used a lot, and he got caught with a punch that Russow hadn't thrown the entire fight and that Duffee hadn't seen coming (followed up by a hammerfist from Satan's own chamber of torture devices, of course). It's the shots you don't see coming that get you, as the saying goes, and he was dominating him badly before that.
Overeem had, literally, just won the K-1 Grand Prix when they fought, and Duffee got knocked out by a K-1 Grand Prix champion. That's not really a terrible thing; he fought a guy that was on a superior level to him at the time and was in over his head.
He was also only 24 when both of those things happened, so it's not like he presented problems during that time that are going to stay with him for the rest of his career, and it's not like being stopped by either of them is a really terrible thing. Look at Overeem; Overeem's been stopped many times (Shogun twice, Liddell, Kharitonov, Arona, Badr Hari, and even Bobby Hoffman all stopped Overeem), and he's still fighting strong and is on the verge of becoming the UFC HW champion.