Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution

Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?

If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.

If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.

This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.

Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”

This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.

Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.

Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.

Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”

Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”

Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”

In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”

Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.

Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”

One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.

Start your day with The Independent, sign up for daily news emails
  • Get to the point
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs General

SThree: Trainee Recruitment Consultant

£18000 - £23000 per annum + Uncapped commission: SThree: Does earning a 6 figu...

Recruitment Genius: SEO Executive

£18000 - £25000 per annum: Recruitment Genius: This is an exciting opportunity...

Recruitment Genius: New Lift Sales Executive - Lift and Elevators

£35000 - £40000 per annum: Recruitment Genius: A challenging opportunity for a...

Recruitment Genius: Customer Service / Receptionist

£14000 - £20000 per annum: Recruitment Genius: This is an exciting opportunity...

Day In a Page

The saffron censorship that governs India: Why national pride and religious sentiment trump freedom of expression

The saffron censorship that governs India

Zareer Masani reveals why national pride and religious sentiment trump freedom of expression
Prince Charles' 'black spider' letters to be published 'within weeks'

Prince Charles' 'black spider' letters to be published 'within weeks'

Supreme Court rules Dominic Grieve's ministerial veto was invalid
Distressed Zayn Malik fans are cutting themselves - how did fandom get so dark?

How did fandom get so dark?

Grief over Zayn Malik's exit from One Direction seemed amusing until stories of mass 'cutting' emerged. Experts tell Gillian Orr the distress is real, and the girls need support
The galaxy collisions that shed light on unseen parallel Universe

The cosmic collisions that have shed light on unseen parallel Universe

Dark matter study gives scientists insight into mystery of space
The Swedes are adding a gender-neutral pronoun to their dictionary

Swedes introduce gender-neutral pronoun

Why, asks Simon Usborne, must English still struggle awkwardly with the likes of 's/he' and 'they'?
Disney's mega money-making formula: 'Human' remakes of cartoon classics are part of a lucrative, long-term creative plan

Disney's mega money-making formula

'Human' remakes of cartoon classics are part of a lucrative, long-term creative plan
Lobster has gone mainstream with supermarket bargains for £10 or less - but is it any good?

Lobster has gone mainstream

Anthea Gerrie, raised on meaty specimens from the waters around Maine, reveals how to cook up an affordable feast
Easter 2015: 14 best decorations

14 best Easter decorations

Get into the Easter spirit with our pick of accessories, ornaments and tableware
Paul Scholes column: Gareth Bale would be a perfect fit at Manchester United and could turn them into serious title contenders next season

Paul Scholes column

Gareth Bale would be a perfect fit at Manchester United and could turn them into serious title contenders next season
Inside the Kansas greenhouses where Monsanto is 'playing God' with the future of the planet

The future of GM

The greenhouses where Monsanto 'plays God' with the future of the planet
Britain's mild winters could be numbered: why global warming is leaving UK chillier

Britain's mild winters could be numbered

Gulf Stream is slowing down faster than ever, scientists say
Government gives £250,000 to Independent appeal

Government gives £250,000 to Independent appeal

Donation brings total raised by Homeless Veterans campaign to at least £1.25m
Oh dear, the most borrowed book at Bank of England library doesn't inspire confidence

The most borrowed book at Bank of England library? Oh dear

The book's fifth edition is used for Edexcel exams
Cowslips vs honeysuckle: The hunt for the UK’s favourite wildflower

Cowslips vs honeysuckle

It's the hunt for UK’s favourite wildflower
Child abuse scandal: Did a botched blackmail attempt by South African intelligence help Cyril Smith escape justice?

Did a botched blackmail attempt help Cyril Smith escape justice?

A fresh twist reveals the Liberal MP was targeted by the notorious South African intelligence agency Boss