It's a well-known principle in economics, however, that price-fixing cartels tend to be unstable and temporary. Why? Among other reasons, the diverging interests of a cartel's various members typically fracture it over time. Similarly, criminal and other conspiracies generally falter — and the bigger they are, the more likely this is — as time passes and the individual conspirators seek to further their own interests, or to minimize risks or harm, by going their separate ways.
This is one of the reasons it's so difficult to view all, most, or even, really, any of Joseph Smith's early associates as conspiring with him to commit fraud. But the principal reason is that there is no substantial evidence to suggest that these early associates were people of bad character, while there is plenty to the contrary.
That should show those critics. They all stuck together and were of good character.