Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution

Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?

If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.

If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.

This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.

Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”

This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.

Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.

Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.

Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”

Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”

Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”

In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”

Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.

Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”

One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.

Start your day with The Independent, sign up for daily news emails
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs General

Recruitment Genius: Web Developer - Junior / Mid Weight

£15000 - £25000 per annum: Recruitment Genius: To support their continued grow...

Recruitment Genius: Marketing Data Specialist

£22000 - £25000 per annum: Recruitment Genius: They are the go-to company for ...

Recruitment Genius: Search Marketing Specialist - PPC / SEO

Negotiable: Recruitment Genius: This is an opportunity to join the UK's leadin...

Recruitment Genius: Sales Administrator

Negotiable: Recruitment Genius: This caravan dealership are currently recruiti...

Day In a Page

Is this the future of flying: battery-powered planes made of plastic, and without flight decks?

Is this the future of flying?

Battery-powered planes made of plastic, and without flight decks
Isis are barbarians – but the Caliphate is a dream at the heart of all Muslim traditions

Isis are barbarians

but the Caliphate is an ancient Muslim ideal
The Brink's-Mat curse strikes again: three tons of stolen gold that brought only grief

Curse of Brink's Mat strikes again

Death of John 'Goldfinger' Palmer the latest killing related to 1983 heist
Greece debt crisis: 'The ministers talk to us about miracles' – why Greeks are cynical ahead of the bailout referendum

'The ministers talk to us about miracles'

Why Greeks are cynical ahead of the bailout referendum
Call of the wild: How science is learning to decode the way animals communicate

Call of the wild

How science is learning to decode the way animals communicate
Greece debt crisis: What happened to democracy when it’s a case of 'Vote Yes or else'?

'The economic collapse has happened. What is at risk now is democracy...'

If it doesn’t work in Europe, how is it supposed to work in India or the Middle East, asks Robert Fisk
The science of swearing: What lies behind the use of four-letter words?

The science of swearing

What lies behind the use of four-letter words?
The Real Stories of Migrant Britain: Clive fled from Zimbabwe - now it won't have him back

The Real Stories of Migrant Britain

Clive fled from Zimbabwe - now it won’t have him back
Africa on the menu: Three foodie friends want to popularise dishes from the continent

Africa on the menu

Three foodie friends want to popularise dishes from the hot new continent
Donna Karan is stepping down after 30 years - so who will fill the DKNY creator's boots?

Who will fill Donna Karan's boots?

The designer is stepping down as Chief Designer of DKNY after 30 years. Alexander Fury looks back at the career of 'America's Chanel'
10 best statement lightbulbs

10 best statement lightbulbs

Dare to bare with some out-of-the-ordinary illumination
Wimbledon 2015: Heather Watson - 'I had Serena's poster on my wall – now I'm playing her'

Heather Watson: 'I had Serena's poster on my wall – now I'm playing her'

Briton pumped up for dream meeting with world No 1
Wimbledon 2015: Nick Bollettieri - It's time for big John Isner to produce the goods to go with his thumping serve

Nick Bollettieri's Wimbledon Files

It's time for big John Isner to produce the goods to go with his thumping serve
Dustin Brown: Who is the tennis player who knocked Rafael Nadal out of Wimbeldon 2015?

Dustin Brown

Who is the German player that knocked Nadal out of Wimbeldon 2015?
Ashes 2015: Damien Martyn - 'England are fired up again, just like in 2005...'

Damien Martyn: 'England are fired up again, just like in 2005...'

Australian veteran of that Ashes series, believes the hosts' may become unstoppable if they win the first Test