History often isn’t kind to those who go on the record making predictions. Albert Einstein once said that nuclear energy would never be a thing, while Margaret Thatcher predicted that a woman would never be prime minister in her lifetime. And remember the record executive who said the Beatles had no future in show business?
People are often spectacularly bad at forecasting the future. But they don’t have to be, says Philip Tetlock, a professor at the University of Pennsylvania who has spent decades studying how people make predictions. In a new book, “Superforecasting: The Art and Science of Prediction,” which he co-wrote with journalist Dan Gardner, Tetlock argues that almost anyone can learn to peer into the future.
Earlier in his career, Tetlock conducted a famous 20-year study in which he had a group of experts make a total of around 28,000 predictions about politics, war, economics and other topics over a timeline of one to 10 years. After scoring all of their predictions against what actually happened, Tetlock’s takeaway was that experts were only about as effective at predicting the future as dart-throwing chimpanzees.
That quip got picked up everywhere, Tetlock says. But there was another important finding from the work that many did not hear. While experts on average didn’t make predictions that were much better than chance, there was a small subset of experts that were actually pretty good at making predictions, a group that Tetlock has come to call “superforecasters.”
Tetlock and his research partner Barbara Mellers spent the last few years recruiting more than 20,000 volunteer forecasters to participate in a massive forecasting activity they dubbed "The Good Judgment Project." To figure out the best methods of forecasting the future, Tetlock, Mellers and other researchers watched these forecasters as they made a multitude of predictions about things like the price of gold and protests in Russia.
The Good Judgment Project competed as one of five teams in a forecasting tournament sponsored by American intelligence agencies, who funded the competition in an effort to learn how to make better predictions after the intelligence failures in 2006, when the United States invaded Iraq believing that Saddam Hussein had weapons of mass destruction. Each of the five teams was required to submit forecasts on a broad range of issues every morning from September 2011 to June 2015, creating a vast trove of data about when predictions succeed and why.
After all this research, Tetlock concluded that the superforecasters aren’t necessarily geniuses, math whizzes or news junkies, though all are intelligent and aware. What separates them from everyone else are certain ways of thinking and reasoning that anyone of decent intelligence can learn, Tetlock says — if they’re willing to put in the work.
The potential gain to society from making better predictions is huge and wide-ranging — from companies thinking about managing risk, to average people investing their retirement funds, to Hollywood producers planning their next blockbuster. The U.S. intelligence community in particular spends billions of dollars a year predicting what governments and other actors around the world are likely to do.
In Tetlock’s eyes, better forecasting means not just making better predictions but also gathering data afterward about which predictions were right and wrong. He says this scientific approach to forecasting and analyzing events could do a lot to resolve some of America’s most controversial and polarized debates in economics and politics.
The scientific method
One of the most important things superforecasters do, Tetlock says, is use the power of probability. You can see that in some of the Good Judgment Project's forecasts for 2016, which were given to me by Warren Hatch, a superforecaster who works in finance when he’s not carefully calculating probabilities.
Hatch walked me through an example. As of Jan. 3, the median estimate of their group was that there is a 68 percent chance that Democrats would win the White House in 2016, a 31 percent chance that a Republican would win, and a 1 percent chance of another scenario — though Hatch cautions that these estimates will probably change as the presidential race evolves.
Here are some of their other forecasts for the coming year (these are the median estimates of their group of forecasters as of Dec. 29):
A 2 percent probability that the Fed will raise interest rates at its January meeting and a 59 percent probability of a hike in March.
A 6 percent chance that a country will leave the euro zone by the end of 2016. If no country leaves the euro zone, they foresee a 35 percent chance that the euro touches the dollar by that time; if a country does leave, they see an 87 percent chance that the euro touches the dollar.
Science news in pictures
Science news in pictures
1/20 'Tiny vampires' existed millions of years ago
Scientists have discovered that microscopic 'vampire' amoebae existed hundreds of millions of years ago, and they may have been some of the first predators on Earth. By examining ancient fossils with an electron microscope, paleobiologist Susannah Porter from UC Santa Barbara discovered tiny holes which may have been drilled by vampiric microbes. The tiny creatures are believed to be the ancestors of modern Vampyrellidae amoebae, and punctured holes in their prey before sucking out the contents of their cells
2/20 Kepler 62f
An Earth-like planet orbiting a star 1,200 light years away could have conditions suitable for life, say scientists. Kepler 62f is about 40 per cent larger than the Earth and may possess surface oceans. It is the outermost of five planets circling a star that is smaller and cooler than the sun discovered by the American space agency Nasa's Kepler space telescope in 2013
3/20 Vegetables grow well in soil from Mars
Scientists have taken a leaf out of the script of The Martian by showing how easy it would be to grow your own veg on the Red Planet. In the hit Ridley Scott film, a stranded astronaut played by Matt Damon uses his botanical skills to cultivate potatoes. Now his success has been emulated by researchers in the Netherlands who harvested tomatoes, peas, rye, rocket, radish and cress raised on simulated Martian soil supplied by Nasa
4/20 Ancient Roman 'leisure complex' unearthed in Jerusalem
An ancient Roman estate complete with its own wine press and bathhouse has been unearthed in Jerusalem. A series of buildings dating back at least 1,600 years were discovered underneath the city's famous Schneller Orphanage which operated on the site from 1860 until the end of the Second World War, when it was turned into an army base. The ruins were discovered by archaeologists from the Israel Antiquities Authority who were excavating the site ahead of building new flats for the city's Orthodox Jewish community
5/20 Scientists discover possible new species of deep-sea octopus nicknamed 'Casper'
Scientists believe they may have found a new species of octopus likened in appearance to Casper, the friendly cartoon ghost. Researchers with the US National Oceanic and Atmospheric Administration made the discovery by chance as they searched the seabed on an unrelated mission collecting geological samples. Teams were operating an unmanned submarine on the Pacific Ocean floor at depths of more than four kilometres (two-and-a-half miles) in the Hawaiian Islands when they spotted the unusual creature
6/20 Black hole captured eating a star then vomiting it back out
Astronomers have captured a black hole eating a star and then sicking a bit of it back up for the first time ever. The scientists tracked a star about as big as our sun as it was pulled from its normal path and into that of a supermassive black hole before being eaten up. They then saw a high-speed flare get thrust out, escaping from the rim of the black hole. Scientists have seen black holes killing and swallowing stars. And the jets have been seen before.But a new study shows the first time that they have captured the hot flare that comes out just afterwards. And the flare and then swallowed star have not been linked together before
7/20 'Male and female brains' aren't real
Brains cannot be categorised into female and male, according to the first study to look at sex differences in the whole brain. Specific parts of the brain do show sex differences, but individual brains rarely have all “male” traits or all “female” traits. Some characteristics are more common in women, while some are more common in men, and some are common in both men and women, according to the study
8/20 Dog-sized horned dinosaur fossil found shows east-west evolutionary divide in North America
A British scientist has uncovered the fossil of a dog-sized horned dinosaur that roamed eastern North America up to 100 million years ago. The fragment of jaw bone provides evidence of an east-west divide in the evolution of dinosaurs on the North American continent. During the Late Cretaceous period, 66 to 100 million years ago, the land mass was split into two continents by a shallow sea. This sea, the Western Interior Seaway, ran from the Gulf of Mexico to the Arctic Ocean. Dinosaurs living in the western continent, called Laramidia, were similar to those found in Asia
9/20 Asteroid to skim past Earth on Halloween 2015
A huge asteroid is set to skim by Earth on Halloween, just three weeks after it was first spotted. The rock is travelling through space at 78,000 miles per hour, and will fly past the Earth at a distance of only 300,000 miles – only slightly further away than our moon, and easily close enough for Nasa to class it a potentially hazardous object. The asteroid is bigger than a skyscraper
10/20 Life on Earth appeared hundreds of millions of years earlier than previously thought
Life may have come to earth 4.1 billion years ago, hundreds of millions of years earlier than we knew. The discovery, made using graphite that was trapped in ancient crystals, could mean that life began "almost instantaneously" after the Earth was formed. The researchers behind it have described the discovery as “a potentially transformational scientific advance”. Previously, life on Earth was understood to have begun when the inner solar system was hit by a massive bombardment from space, which also formed the moon's craters
11/20 Earth could be at risk of meteor impacts
Earth could be in danger as our galaxy throws out comets that could hurtle towards us and wipe us out, scientists have warned. Scientists have previously presumed that we are in a relatively safe period for meteor impacts, which are linked with the journey of our sun and its planets, including Earth, through the Milky Way. But some orbits might be more upset than we know, and there is evidence of recent activity, which could mean that we are passing through another meteor shower. Showers of meteors periodically pass through the area where the Earth is, as gravitational disturbances upset the Oort Cloud, which is a shell of icy objects on the edge of the solar system. They happen on a 26-million year cycle, scientists have said, which coincide with mass extinctions over the last 260-million years
12/20 Genetically-engineered, extra-muscular dogs
Chinese scientists have created genetically-engineered, extra-muscular dogs, after editing the genes of the animals for the first time. The scientists create beagles that have double the amount of muscle mass by deleting a certain gene, reports the MIT Technology Review. The mutant dogs have “more muscles and are expected to have stronger running ability, which is good for hunting, police (military) applications”, Liangxue Lai, one of the researchers on the project. Now the team hope to go on to create other modified dogs, including those that are engineered to have human diseases like muscular dystrophy or Parkinson’s. Since dogs’ anatomy is similar to those of humans’, intentionally creating dogs with certain human genetic traits could allow scientists to further understand how they occur
13/20 Nasa confirms Mars water discovery
Nasa has announced that it has found evidence of flowing water on Mars. Scientists have long speculated that Recurring Slope Lineae — or dark patches — on Mars were made up of briny water but the new findings prove that those patches are caused by liquid water, which it has established by finding hydrated salts.
14/20 Bees in the Rocky Mountains are evolving shorter tongues
With warmer summers, flowers in the Rockies have become shallower and more suited to shorter-tongued bees
15/20 The majority of the UK public believe in aliens
The titular alien character from 2011's 'Paul' - a poll has found the majority of the public in Britain, Germany and the US believe that intelligent life is out there in the universe
16/20 Researchers discover 'lost world' of arctic dinosaurs
Scientists say that the new dinosaur, known as Ugrunaaluk kuukpikensis, “challenges everything we thought about a dinosaur’s physiology”. Florida State University professor of biological science Greg Erickson said: “It creates this natural question. How did they survive up here?”
17/20 Scientists find exactly what human corpses smell like
New research has become the first to isolate the particular scent of human death, describing the various chemicals that are emitted by corpses in an attempt to help find them in the future. The researchers hope that the findings are the first step towards working on a synthetic smell that could train cadaver dogs to be able to more accurately find human bodies, or to eventually developing electronic devices that can look for the scent themselves.
18/20 The Syrian civil war has caused the first ever withdrawal from the 'doomsday bank'
Researchers in the Middle East have asked for seeds including those of wheat, barley and grasses, all of which are chosen because especially resistant to dry conditions. It is the first withdrawal from the bank, which was built in 2008. Those researchers would normally request the seeds from a bank in Aleppo. But that centre has been damaged by the war — while some of its functions continue, and its cold storage still works, it has been unable to provide the seeds that are needed by the rest of the Middle East, as it once did.
19/20 A team of filmmakers in the US have made the first ever scale model of the Solar System in a Nevada desert
Illustrations of the Earth and moon show the two to be quite close together, Mr Overstreet said. This is inaccurate, the reason being that these images are not to scale.
20/20 Academics claim a full bladder makes for a better liar
People lie more convincingly if they have a full bladder, according to research by academics at California State University. Iris Blandón-Gitlin's team asked 22 students to lie to a panel of interviewers. Half were given 700ml to drink before the interview and the other half, just 50ml. The students with the full bladders showed fewer signs that they were lying and their untrue answers were longer and more detailed, meaning interviewers were less able to detect that they were telling porkies. PM David Cameron has previously attested to giving speeches on a full bladder.
An 8 percent chance that Congress will ratify the Trans-Pacific Partnership trade agreement before the 2016 general election, but a 60 percent chance it will be ratified between the election and the end of 2017.
These odds might seem strangely specific, but for the superforecasters they are typical. Gardner and Tetlock write that most people use a “two- or three-setting mental dial,” thinking of probabilities in terms of only “yes,” “no” or “maybe.” But the superforecasters drill down on much more specific probabilities, taking care to consider whether something really has a 62 or 63 percent chance of occurring.
How did the forecasters reach the presidential election figure? Most of us would probably start by thinking about the political climate in the United States and the popularity of Barack Obama, Hillary Clinton, Donald Trump and the other Republican candidates. But Hatch cautions against this approach.
Instead, he and other forecasters try to establish a ballpark figure using history and statistics before they get into the details.
Superforecasters at a conference in London in October 2015. Hatch is in the bottom row, second from the right. Photo credit: Scott Eastman. Superforecasters at a conference in London in October 2015. Hatch is in the bottom row, second from the right. (Scott Eastman)
For example, they might look at how often a president of a different party is elected when the U.S. economy is growing at only slightly above 2 percent, as it is now. Or they might look at how often a president of the same party is elected after a two-term presidency, like that of Obama. (Hatch says it’s about a third of the time.)
Only after creating this estimate will the forecasters use their knowledge of the specific situation to adjust that estimate up or down, Hatch says. They might think about America's changing demographics and Marco Rubio's broad appeal, and bump up the estimate for the Republican Party. Or they might read news stories that attest to Clinton’s momentum and raise the estimate for the Democrats, he says.
This approach echoes the thinking of Daniel Kahneman, a psychologist who studies decision making and won the Nobel prize for his work in behavioral economics. Kahneman calls the broad, statistical estimate that the superforecasters make first "the outside view," and he's cautioned people to be more aware of the outside view when making predictions and decisions.
By doing that, he and others believe people can partially counter the tendency of the human brain to focus on our own personal experiences — what Kahneman calls "the inside view" — and ignore the substantial power of history and statistics to at least provide context.
Start making sense
Another key to being a good forecaster is to try to keep your beliefs from clouding your perceptions. This is a fault that Tetlock and Gardner say is far too common among the talking heads on television and in the newspaper who make the most influential predictions in America, about things like whether the United States can defeat the Islamic State, which candidate will win the Iowa primary, or if the Fed will raise interest rates.
People want answers. As a result, the media gravitate toward those with clear, provocative and easy-to-understand ideas, and audiences tend to believe those who speak with confidence and certainty.
The problem is that those who speak with confidence and certainty and spin a clear narrative are actually less likely to make accurate predictions, say Tetlock and Dan Gardner. And the more famous the expert is, the worse he or she seems to be at forecasting the future. Tetlock’s original study — the one in which he concluded that experts were roughly as effective as chimpanzees — actually showed an inverse relationship between the fame of an expert and the accuracy of their predictions.
People have a tendency to selectively absorb facts that confirm their worldview, and ignore those that don’t — what psychologists call “confirmation bias.” As Kahneman writes, “Declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”
This may be particularly true for people who tend to organize their thinking around a “Big Idea” that they are passionate about — whether socialism, free-market fundamentalism or impending environmental doom. Tetlock dubs these kind of ideologically minded people "hedgehogs" after a phrase from Greek poetry (“The fox knows many things but the hedgehog knows one big thing.”)
How do you spot a hedgehog? Tetlock says to listen to their arguments and watch out for "the experts who say 'moreover' more often than they say 'however'" — i.e., people who double down on their arguments without qualifying them with contrasting information — as well as those who declare things “impossible” or “certain."
You can probably guess that Tetlock's superforecasters are not "hedgehogs." Instead, they are the kind of person that Tetlock dubs "foxes."
According to Tetlock, foxes are more pragmatic and open-minded, aggregating information from a wide variety of sources. They talk in terms of probability and possibility, rather than certainty, and they tend to use words like “however,” “but,” “although” and “on the other hand” when speaking.
When it comes to prediction, the foxes outfox the hedgehogs. While the hedgehogs managed to do slightly worse than random chance, the foxes had real foresight. The foxes weren’t just more cautious in their predictions, they were a lot more accurate.
Unlike hedgehogs, foxes also had a desire to keep reviewing their assumptions, updating their estimates, and improving their understanding — a kind of attitude that Tetlock calls "perpetual beta." Tetlock says that this approach is the single most important ingredient for learning to make accurate predictions. “For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded,” Tetlock and Gardner write.
Testing the results and figuring out which predictions were right and which were wrong is an essential part of Tetlock's formula. Without evaluating the result of a prediction, it's like "practicing free throws in the dark," he says — impossible to get better.
Unfortunately, most of the predictions you see in the media lack the specificity necessary to test them, like a specific time frame or probability, Tetlock says.
"[Experts will] say emphatic sounding things, like the Eurozone is on the verge of collapse, but they’ll say 'The Eurozone may well be on the verge of collapse' — there’s no certainty, and there’s no date. I don’t know what it means to say something 'may well be' — it could be 20% or 80%," he says.
Instead, Tetlock advocates for something he calls "adverserial collaboration" — getting people with opposing opinions in an argument to make very specific predictions about the future in a public setting, so onlookers can measure which side was more correct.
For example, hawks and doves on the Iran nuclear deal, or those who advocate austerity or stimulus policy for Europe's downtrodden economies, could go head to head in making predictions, and let the public evaluate which side is more accurate. Tetlock says this process could help make real progress on some of the most controversial issues of our time, and help depolarize unnecessarily polarized debates.
Tetlock acknowledges that the most powerful and famous experts around will probably have little motivation to join this kind of debate, since they would have little to gain and everything to lose. "The more status you have in the system, the less interested you should be in testing whether you are better than ordinary mortals." Of his forecasting tournaments, he says, "I’m not waiting for Tom Friedman to sign up."
For most mere mortals though, the gains to be had from making more scientific and accurate predictions are immense. “To be sure, in the big scheme of things, human foresight is puny, but it is nothing to sniff at when you live on that puny human scale,” Tetlock and Gardner write.
Washington PostReuse content