The truth about the wisdom of crowds

A century later, economists and statisticians are still arguing over how scattered opinions of ordinary people can produce amazingly accurate insights.

Manchester City fans celebrate a victory, which, according to the "wisdom of crowds" theory, could have been predicted based on the scattered opinions of the football-watching public.

People who claim to see the future are a staple of Hollywood plots. Now their real-life counterparts can test their prowess by trying to predict the next box office smash. Two US financial companies are about to launch futures exchanges allowing everyone from city slickers to cineastes to trade in the success of new movies. The idea is that punters benefit financially from predicting box office receipts, while filmmakers benefit by backing their projects - or shorting them if they're going off the rails.

The companies behind the exchanges, Cantor Fitzgerald and Veriana, hope to capitalise on the familiarity of the market. Few of us may have a view on, say, the 36-month price of light sweet crude, but we're all used to assessing the movies shown in the forthcoming features ads in cinemas. Yet just because something is familiar doesn't make it easy to predict. Who would have predicted the soaraway success of the 1999 horror flick The Blair Witch Project, which took in around $250 million (Dh900 million) at the box office, almost 500 times its budget. In fact, lots of movie fans did - and their success has become a celebrated example of the so-called "wisdom of crowds", according to which the apparently scattered opinions of ordinary people can produce amazingly accurate insights.

The phenomenon was first investigated a century ago by the English polymath Francis Galton after attending a country fair in Plymouth. He came across a guessing game in which, on payment of a small entry fee, people could attempt to estimate the weight of an ox. Needless to say, most of the 700-odd guesses were wide of the true weight of 543kg. Yet when Galton examined the spread of the estimates, he found that the median value was 547.5 kg - within one per cent of the true value.

How had the collective estimate of the crowd proved so reliable? Some of those taking part were farmers and other livestock professionals, but their expert judgements would have been swamped by those of the rank amateurs. Galton himself suggested the answer might partly lie in the entrance fee which, though small, might help deter time wasters. A century later, economists and statisticians are still arguing over how the wisdom of crowds works. At least now they have plenty of data, gathered from the host of "prediction markets" now available online, many of which are impressively reliable.

One of the first and most trusted is the Iowa Electronic Market (IEM) set up in 1988 by academics at the University of Iowa to study wisdom of crowds effects. The IEM is best known for trading in imaginary securities centred on the outcome of political events. And over the years, the ebb and flow of the "share prices" of candidates have proved reliable indicators of the outcome of US presidential elections. Indeed, they have often turned out to be better than conventional opinion polls - even predicting the outcome of the notoriously close US presidential contest between George W Bush and Al Gore in 2000.

Today it is possible to find online prediction markets for everything from the likelihood of Barack Obama, the US president, being re-elected in 2012 (currently estimated at a paltry 37 per cent on the news prediction website hubdub.com), to the chances of Tiger Woods playing a PGA tour event before the end of April (which pundits at intrade.com currently rate at better than 80 per cent). Analysis of data from these markets suggests Galton was right. While many of the prediction markets don't involve real money, punters still face an "entrance fee" plus the risk of losing kudos, as measured by their ranking in league tables. And that, it seems, is enough to compel people to do a good job.

Kudos alone has been enough to give the Hollywood Stock Exchange (HSX) its reputation for reliable predictions of the fate of new movies (the success of The Blair Witch Project among them). It is also pretty good at predicting both the nominees and the winners of the Oscars. This year was no exception, with HSX traders correctly forecasting nine of the 10 titles nominated for Best Picture, and six out of eight winners of the principal Oscars.

Conventional market research companies are not impressed, of course. They point out that their polls are based on the only mathematically proven method for minimising biased results: random sampling. And unlike prediction markets, their polling results come with estimates for the likely margin of error. Not that this has stopped them coming up with risibly poor predictions over the years. Certainly, the lack of a detailed theory underpinning prediction markets hasn't stopped some major companies using them to forecast sales, among them Hewlett-Packard, Google and Merck.

But is it even worth trying to understand the success of prediction markets? Perhaps not, given the evidence now emerging that forecasting may be easier than people think. Dr Sharad Goel and colleagues at Yahoo! Research in New York have examined the success of various forecasting methods, and confirmed that predictive markets do tend to outperform the rest. But they also found that the margin wasn't all that great - and that much simpler techniques can do almost as well.

For example, the ability of the HSX prediction market to predict box office revenues on opening weekends is often cited as evidence of its uncanny power. Using data for around 100 movies, the team found that HSX deserved its reputation. Yet the team also found that they could do almost as well using a simple rule based on just two commonsense bellwethers: the level of web searches for the movie in the run-up to its premiere, plus the number of screens slated to show it when it opens.

Just before the opening last December of the blockbuster Avatar, team member Dr Duncan Watts used the simple rule to forecast the film's revenues over the first weekend. He put the figure at somewhere in the range $65 million (DhD240 million) to $84 million. The actual figure was slap in the middle: $77 million. Anyone hoping to use the team's discovery to make money on the upcoming Hollywood futures exchange should beware, however. It is hard to keep clever insights under wraps for long. So don't go telling everyone about it - and good luck.

Robert Mathews is visiting reader in science at Aston University, Birmingham, England.