Humans have flawed brains that cause them to act crazy sometimes. And in groups, people get even more crazy. Many smart people believe dumb things. Sometimes a group of otherwise completely sane people come together and do something insane. History is full of examples of the “ Extraordinary Popular Delusions and the Madness of Crowds, or Manias, Panics and Financial crisis. Humans sometimes join suicide cults. Human literally burned witches not that long ago. Groupthink is a helluva drug.

Financial markets provide an arena in which the biggest gains can be made betting against consensus. Yet statistically speaking, the consensus is usually right. The times when the crowd goes crazy are notable because they are exceptions. One must carefully decide when to be a contrarian.

Under what conditions is the consensus likely to be wrong? Lets invert the question: under what conditions is the crowd likely to be right?

Most of the time, a large group of people actually comes to a more accurate conclusion than any one individual. The success of Estimize, which crowdsources earnings estimates is one useful empirical example. “Wisdom of Crowds” is a well documented phenomenon, and well summarized in the book by the same title.

Why is the crowd so often right? What must happen for the crowd to be right ? Researchers have identified four interrelated conditions that encourage the wisdom of the crowds:Diversity, Information Availability, Decentralization, and existence of an Aggregation Mechanism.

Understanding these conditions can help one know when to follow the zeitgeist, and when to make a contrarian bet against it. A firm grasp of the facts on both sides of a controversy is necessary, but possibly not sufficient. One can never be sure of all the facts. Its also useful to understand the broader social forces, and how they influence the likelihood of the consensus being right or wrong. Searching for these conditions(or their absence), can be a useful method of avoiding cults and identifying opportunity, beyond just the facts of the individual situations.

The biggest gains are available by being a contrarian who understands the crowd.

The key is understanding when the wisdom of crowds flips to the madness of crowds. And the essential insight is that it has to do with a violation of one or more of the core conditions for a wise crowd.

Michael Mauboussin: Who is on the other side?

Diversity

Diversity implies that each person has their own point of view and some private information, even if only their unique interpretation of the available public information. Diversity is important because it adds different perspectives and increases the amount of available information.

The Value of Crowdsourcing: Evidence from Earnings Forecasts

Crowded trades have a tendency to crash- leading to no bid markets all the way down. Academics have noted that in a run up to a market crash diversity of population declines. The market becomes fragile, and eventually there is no one to buy from (or sell to).

One reason Estimize’ earnings estimates have tended to be better than Wall Street Sell side is that Estimize analysts are a diverse group of independent thinkers from around the world, holding a variety of different jobs. A lot of Wall Street analysts go to the same conferences, went to the same schools, etc.

When people come to the same conclusion from different backgrounds, logical methods, etc , their collective wisdom refines understanding of reality. The opposite occurs when people feel pressure to conform. Its important to note this is a genuine deep diversity of thought and perspective, not a superficial check the box diversity.

If multiple people with different viewpoints all come to similar conclusions, the odds of the opposite being true decrease substantially. On the other hand, if there is an obivous archetype of the thinker on the other side, then maybe a contrarian opportunity is available.

As Michael Maubossin has noted- conformity is a nonlinear process:

Scientists even have a sense of the neurobiological basis for conformity.Informational cascades occur when individuals follow the decisions of those who precede them without regard to their personal information.

Epidemiological models are useful here.

Information

The wisdom of crowds does not emerge in groups of idiots. It only applies when there is widespread access to information necessary to reach a conclusion. The extreme opposite occurs in totalitarian societies (or large corporations), where information is tightly restricted. Prior to the internet, information sometimes diffused slowly through a market, leading to massive price discrepancies obvious at even a quick quantitative glance.

Quality of input is critical to the success of crowdsource analysis:

Alternatively, it is possible that the inclusion of forecasts from certain individuals, such as Non-Professionals, may provide no value, or worse, cause the Estimize consensus to deviate further from actuals. Surowiecki (2004) states that although diversity matters, assembling a group of diverse but thoroughly uninformed people is not likely to lead to wise outcomes.

Independence

Independence is related to diversity. People need to be able to freely analyze reality and discuss opinions. Conventional wisdom is more likely to be accurate when it is freely subjected to challenge. When there are institutional or social factors that make people extremely afraid to speak truth, what everybody says to be true, may be wrong.

Children learn of this phenomenon early through the Emperor Wears No Clothes. For grown ups, see Death of Stalin for an example of lack of independence leading to morbidly hilarious results.

Independence requires relative freedom from opinions and actions of others, not complete isolation. Independence enables people to actually express their diverse information and reduces potential bias in the group decision.

Decentralization

Decentralization allows people to specialize and draw on local knowledge, without any individual or small group dictating the process.

Diversity and independence all fit in nicely with decentralization. Through specialization, decentralization encourages independence and increases the scope and diversity of information. Decentralization reduces the risk that independence and diversity will go away. Similarly having capital flows from all around the world, not just from a small group of schools or similarly thinking firms, increases the likelihood that markets become more efficient.

Existence of an Aggregation Mechanism

Finally, an aggregation mechanism is necessary to collect the individual opinions and harness the ‘wisdom-of-crowds’ effect.

This is basically why capitalism has succeeded. The price mechanism aggregates facts about supply and demand better than any bureaucracy could. At the same time, this why often the best opportunities to earn an investment profit are in illiquid asset classes where the market does not function as an aggregation mechanism to make the price close to right.

Of course, just because the market consensus is wrong, doesn’t mean that is necessarily wise to bet against it today. Must also consider reflexivity, narratives, and capital flows etc, and maintain a balance sheet that allows one to survive long periods of mass delusion.

Postscript: This is all indirectly related earlier post on finding underfollowed opportunities: The hard thing about finding easy things . This linked the ideas of both Sun Tzu and Warren Buffett. Some of the specific opportunity sets mentioned in this post have since been too widely known and we’ve moved further into more esoteric off the beaten path ideas. Nonetheless the basic principal still holds: there are more likely to be opportunities where the crowd isn’t looking.