Prediction Markets as a Tool for Intelligence Analysis

The raison d’être for intelligence analysis is being able to predict near-future events with reasonable accuracy. Based on the little that I have read, there is no mechanism that seeks to improve Indian intelligence analysts’ accuracy over time. This leads to what Philip Tetlock calls ‘outcome-irrelevant learning’.  This is a situation wherein no matter what happens in reality, people are in an excellent position to explain that what happened was consistent with their own view.

Outcome-irrelevant learning is quite easy to find in mainstream foreign policy analysis. Many analysts would, for example, argue that talks between India and Pakistan at the highest levels are a necessary policy instrument for managing Pakistan. When presented with the evidence to the contrary they would still be in a position to come up with a reason that deflects blame from their policy proposal.

The costs of outcome-irrelevant learning become very high if intelligence analysis also falls into this same trap.  It’s all the more necessary in that community to make people remember their previous states of ignorance and make them update their Bayesian priors when things don’t go according to their expectations.

This is where a prediction market comes in. In such a market, a group of people speculate on future events. Each individual assigns a probability to a near-future event and these choices get registered. Once the event actually takes place, you get a chance to reflect on why you were wrong (or right). By looking at the track records of analysts over time for several questions, one can wean out the worse analysts from the better ones.

The US intelligence community recognised the value of these markets more than ten years ago. This CIA paper from 2006 concludes:

The record of prediction markets is impressive. For the US Intelligence Community, prediction markets offer a method by which to improve analytical outcomes and to address some of the deficiencies in analytical processes and organisation. In the realm of intelligence analysis, prediction markets can contribute to more accurate estimates of long-term trends and threats and better cost-benefit assessments of ongoing or proposed policies.

It’s time that we introduce such prediction markets to the Indian intelligence community as well. To improve prediction skills, the training programme for new recruits can include elements from Tetlock and Gardner’s classic Superforecasting.

We recently authored a paper that looks at human resource reforms for India’s external intelligence agency. We should have given thought to prediction markets, particularly on the section related to the training of intelligence officers.