Seminars About Long-term Thinking: Philip Tetlock: Superforecasting

November 26, 2015 9:36 AM

The pundits we all listen to are no better at predictions than a "dart-throwing chimp," and they are routinely surpassed by normal news-attentive citizens. So Philip Tetlock reported in his 02005 book, Expert Political Judgement—and in a January 02007 SALT talk. It now turns out there are some people who are spectacularly good at forecasting, and their skills can be learned. Tetlock discovered them in the course of building winning teams for a tournament of geopolitical forecasting run by IARPA—Intelligence Advanced Research Projects Activity. His brilliant new book, SUPERFORECASTING: The Art and Science of Prediction, spells out the methodology the superforecasters developed. Like Daniel Kahneman's THINKING, FAST AND SLOW, the book changes how we think about thinking.
posted by eotvos (3 comments total)

This was an interesting talk. But, the specific claim that US intelligence agencies who were wrong about Iraq possessing weapons of mass destruction in 2002 were *mistaken* and not *lying* seems unreasonably credulous. Given that nearly every reputable information source at the time not directly involved in the US war effort expressed extreme skepticism, repeating the government line wholesale without any hesitation makes it hard to take the speaker seriously when he talks about subjects about which I know less. The temptation to write him off as an organ grinder's monkey for the DOD is hard to resist.

Also, winning a one-off contest is pretty far from a controlled experiment. I'm willing to be convinced that this is science, but it sure hasn't happened in this talk. I'm astonished nobody asked "is it reproducible" during questions. Better than the status quo isn't nearly good enough when the consequences are thousands of lives and billions of dollars. This whole multi-million dollar effort sounds like the thing one normally does *before* writing a proposal in any discipline in which it is possible to be proven wrong. In short, everything here fails the sniff test in a big way.
posted by eotvos at 12:31 PM on November 26, 2015

I'll check this out later, but can you tell me if they cover The Good Judgement Program, as covered in NPR back in 2014? That sounded like a more robust and long-term study than a one-off event where people might have simply gotten lucky, as you mention.

Based on the current website (GoodJudgementProject.com bounces you to GJOpen.com), the program now appears to have shifted from a long-term study on the ability of "regular people" with only access to world news (instead of CIA-type secret information) to one of an open contest to possibly find more "superforecasters" (or just drum up interest in "open intelligence," as one open question is currently "Will Donald Trump win Iowa's Republican caucus?")
posted by filthy light thief at 2:41 PM on November 27, 2015

No mention of the Good Judgment project specifically, unless I missed it. (I was doing other things while listening.) But it sounds pretty interesting - thanks for pointing it out.
posted by eotvos at 3:42 PM on November 27, 2015