Free sample lesson · Superforecasting: The Art and Science of Prediction
Why Confident Experts Miss
Retirees and volunteers beat analysts with classified files. What did they do differently?
Lesson 1 of the Wondering course Superforecasting: The Art and Science of Prediction
A Confident Call Gone Wrong
In October 2002, the National Intelligence Council officially judged that Iraq had chemical and biological weapons.[[8]](https://hbr.org/2016/05/superforecasting-how-to-upgrade-your-companys-judgment)
The judgment was wrong. The shock pushed the U.S. intelligence community to look for better ways to forecast.[[8]](https://hbr.org/2016/05/superforecasting-how-to-upgrade-your-companys-judgment)
Its research agency, IARPA, ran a forecasting tournament on national security questions.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/) Five university programs competed on roughly 500 questions from 2011 to 2015, making over one million judgments.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
After two years, the Good Judgment Project was winning by such margins that IARPA put the remaining money into it.[[5]](https://www.econtalk.org/philip-tetlock-on-superforecasting/) So who was on that team?
- Wrong judgment: In 2002 the official Iraq weapons judgment proved wrong.
- Search begins: The intelligence community looked for better forecasting.
- Tournament: IARPA posed about 500 questions to five university programs.
- Clear winner: The Good Judgment Project won by wide margins and got the remaining money.
Retirees Beat Classified Files
Bill Flack was a 56-year-old retiree in rural Nebraska who had worked for the Department of Agriculture. He often did not read a newspaper.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
Mary Simpson, an economist, had missed the 2007-2008 financial crash.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
The publisher describes tens of thousands of ordinary volunteers, including a filmmaker, a retired pipe installer and a former ballroom dancer.[[1]](https://www.penguinrandomhouse.com/books/227815/superforecasting-by-philip-e-tetlock-and-dan-gardner/) The best beat even intelligence analysts with classified information.[[1]](https://www.penguinrandomhouse.com/books/227815/superforecasting-by-philip-e-tetlock-and-dan-gardner/)
Each year the top two percent, about 60 people, were named superforecasters.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/) They were not paid for being right. Each volunteer got a flat amount at the end of each tournament year, the same whether their forecasts were good or bad: $150 after the first year, $250 after the second.[[6]](https://escholarship.org/content/qt4rg4n9vr/qt4rg4n9vr.pdf)
- Share named superforecasters: Chosen each year from the volunteer pool.
- Superforecasters per year: A title earned by record, not by job.
- Pay after Year 1: The same whether forecasts were good or bad.
- Pay after Year 2: Money was not what drove accuracy.
Confidence Is Not Accuracy
Before the tournament, Tetlock studied nearly 300 experts, about two-thirds with Ph.D.s.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
Confidence is how sure someone says they are. Accuracy is how often their calls come true. They are separate measurements.
Tetlock reports that when experts said they were 80 or 90 percent confident, they were often right only 60 or 70 percent of the time.[[2]](https://www.edge.org/conversation/philip_tetlock-a-short-course-in-superforecasting)
- Stated confidence
- Actual accuracy
A sure tone told you little about the record.
Hindsight Feels Like Foresight
Almost nobody predicted where Gorbachev would take the Soviet Union.[[2]](https://www.edge.org/conversation/philip_tetlock-a-short-course-in-superforecasting)
Afterward, nearly every expert had an explanation that fit their prior worldview.[[2]](https://www.edge.org/conversation/philip_tetlock-a-short-course-in-superforecasting)[[5]](https://www.econtalk.org/philip-tetlock-on-superforecasting/) Tetlock borrows a phrase that another psychologist coined: an "outcome irrelevant learning situation". Whatever happens, the expert feels right.
A fluent story after the event is not foresight. Almost any outcome can be explained afterward.
This does not mean experts know nothing. A popular summary of Tetlock's expert study says the experts did no better than a chimp throwing darts, that is, no better than random guessing, or "chance". That line is a joke, not the finding: some forecasters, the modest foxes you will meet next, did beat chance in the short run. What set them apart?
Foxes and Hedgehogs
Tetlock borrowed two animals from Isaiah Berlin's essay "The Hedgehog and the Fox".[[2]](https://www.edge.org/conversation/philip_tetlock-a-short-course-in-superforecasting)
A hedgehog fits every event into one big theory and speaks with certainty. Tetlock names the defining trait of a bad forecaster: "Dogmatism".[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
A fox weighs many small ideas, stays self-critical and asks what would prove them wrong.
Modest, self-critical foxes beat confident hedgehogs.[[2]](https://www.edge.org/conversation/philip_tetlock-a-short-course-in-superforecasting) For example, "My theory explains it all" sounds like a hedgehog. "Here is what would change my mind" sounds like a fox.
- Hedgehog
- Fox
Grand Slams vs Batting Average
In 1992, a hedgehog predicted a Russia-Ukraine war by 1997.[[3]](https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-i)
1997 passed with no such war. Something like it came in 2014, so the hedgehog could later claim to have "seen it coming".
Scored on its own five-year window, the forecast failed badly.
- Hedgehog makes forecast: Predicts a Russia-Ukraine war by 1997.
- Deadline passes: No such war inside the five-year window, so the forecast fails.
- Something like it: Years late, yet it lets the hedgehog claim to have seen it coming.
Tetlock notes that hedgehogs are sometimes spectacularly right: the biggest single hits in forecasting tournaments tend to go to them. Foxes do better across their whole record. He puts it in baseball terms: hedgehogs hit more grand slams; foxes have the better batting average.[[3]](https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-i) One famous early call is an anecdote. So which record would you rather trust?
Habits You Can Learn
So what did Flack and the others do differently? Flack did not treat a Wikipedia article as the final word; he used it as a starting point for finding other sources, or in his words, "as a source of sources".[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
With constant feedback, superforecasters checked whether they had been over- or underconfident. Of good results, they asked, "Were we lucky?"[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
- Make a forecast: Gather sources and commit to a call.
- Question resolves: The outcome becomes known.
- Check confidence: Were you over- or underconfident?
- Ask: were we lucky?: A good result is not automatically good judgment.
The book's claim is that foresight comes from habits of thought that can be learned.[[4]](https://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future/)
But keeping score needs a checkable forecast. The 2002 Iraq judgment was stated as near-certain fact, with no number attached. How would you score a call like that?
Check yourself
A forecaster compares several small ideas, asks what would change their mind, and avoids certain language. This person is a _____.
- fox
- hedgehog
Answer: fox
A fox weighs many small ideas and stays self-critical. A hedgehog relies on one big theory and speaks with certainty.
After a forecast resolves, strong forecasters first _____ whether they were over- or underconfident. After a good result, they also ask whether they were _____.
Answer: After a forecast resolves, strong forecasters first check whether they were over- or underconfident. After a good result, they also ask whether they were lucky.
Checking confidence helps reveal calibration problems, and asking whether success was luck prevents one good result from creating false confidence.
An expert says a forecast is 90% likely, but similar forecasts have come true only 60% of the time. What should you conclude?
- The forecast is probably correct because 90% is high.
- The expert may be overconfident, so check the person’s record.
- The forecast must be wrong because experts cannot predict anything.
- The expert’s explanation after the outcome will show whether it was accurate.
Answer: The expert may be overconfident, so check the person’s record.
The expert’s stated confidence and actual accuracy are separate. A confident tone is poor evidence unless the person’s record supports it.
Keep going
This is lesson 1 of 11 in Superforecasting: The Art and Science of Prediction. Next up:
- Turn Vague Words into Numbers
- Scoring Forecasts with Brier
- Calibration Versus Resolution
- Learning from Hits and Misses
- Start with the Base Rate
Wondering turns any topic into short lessons you can read, watch, or listen to, with practice questions and an AI tutor.