The isolation is a big part of it. Joseph Smith was young, handsome, charismatic, and rhetorically skilled (though not well educated). He revived the faith of a group of people who had been taken in con job after con job by charlatans. Somehow he seemed different to them, and he wasn't fucking around- this guy was willing to move anywhere and do anything to build his church, and making lots of enemies in the process held his people together, them against the world. I guess you can never tell when a martyrdom will stick to the ribs, but they loved him enough that it really stuck.
One of the other big things that kept them together was the promise that they, like Adam, would be exalted and reborn as a creator not unlike God or Jesus, to rule over their own celestial piece of real estate. Their time on Earth was a test of faith (a common sort of belief) and a judgment of their good works, but their end was not just to rejoin the creator, but to become Him. The better you kept up with your payments to the Church the more secrets would be revealed, and it still works like that today, though the Internet has taken almost all of the secrecy out of it.
So he sold them a gigantic promise with a great sales pitch, and he was pretty legit in terms of being the pioneer type and the type who would build your house alongside you. Brigham Young was probably lesser in the charismatic leadership role but had business acumen and was good with logistics and hardman leadership. He also must have been quite smart, since he led the building of a successful society that was under serious pressure to survive their first year, and set up great city layouts (best thing about SLC imo). Settling on the Wasatch Front was also a clever or serendipitous move since it was very harsh in every direction for hundreds of miles, so people were inclined to stay after the difficult journey there.
I dunno, just some random ramblings.