Book Summary: Superforecasting

Book: Superforecasting

Author: Philip Tetlock

Things to consider when forecasting for the 1st time: 

• Outside view: another way of thinking about this is “what is the base case?”
• Problem Substitution: given that forecasting is hard and our brains like shortcuts, are you solving for a problem that’s different from what’s being asked. (example given in the book is “Will Abe visit Yasukani?” vs. “If I were the PM of Japan, would I visit Yakusani?”).
• Reasonable Precision in Forecasting: It is important to speak of probabilities in numerical terms (even if approximate) vs. speaking in words, because words such as “probable” and “certain” mean very different things to different people. “I’d rather be a bookies than a goddamn poet” – Sherman Kent.
• Check for Scope Sensitivity: The author cites an example of fall of Assad regime, where regular forecasters had 40% chance of the regime falling in 3 months and 41% in 6 months, whereas super-forecasters put chances of 15% over 3 months and 24% over 6 months. This suggests that the estimate is much more thought-through, and when making ests. we should check for scope-sensivities to see if we’re making good calls, or just picking some sort of a representative number coming thru a simple bias somewhere.

Things to consider when maintaining a forecast: 

• Forecast -> Measure -> Revise: These are often used only within the rare confines, and should be more commonplace.
• Bayesian updates: modest revisions are much more preferred as opposed to wide swings in opinion.
• Frequency: super-forecasters update often

Things to consider when doing a post-mortem: 

• How much is luck vs. skill
• What went wrong?: very critical to understand where the forecast went wrong, to understand what needs correcting.
• Avoid Forer Effect: we tend to kid ourselves, and are pleased with revisionist history of vague forecasts (hence,  the importance of precision/numeracy in forecasts). According to a study by the author, within 1-2 years, experts remembered their predictions as 31% more predictive than they actually were.