Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution

Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?

If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.

If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.

This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.

Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”

This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.

Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.

Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.

Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”

Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”

Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”

In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”

Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.

Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”

One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.

Start your day with The Independent, sign up for daily news emails
Voices
Lucerne’s Hotel Château Gütsch, one of the lots in our Homeless Veterans appeal charity auction
charity appeal
Arts and Entertainment
Tony Hughes (James Nesbitt) after his son Olly disappeared on a family holiday in France
tv
News
people

Jo from Northern Ireland was less than impressed by Russell Brand's attempt to stage a publicity stunt

Sport
Scunthorpe goalkeeper Sam Slocombe (left) is congratulated by winning penalty taker Miguel Llera (right)
football
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
News
ebooksNow available in paperback
Arts and Entertainment
The Apprentice candidates Roisin Hogan, Solomon Akhtar, Mark Wright, Bianca Miller, Daniel Lassman
tvReview: But which contestants got the boot?
Life and Style
A woman walks by a pandal art installation entitled 'Mars Mission' with the figure of an astronaut during the Durga Puja festival in Calcutta, India
techHow we’ll investigate the existence of, and maybe move in with, our alien neighbours
Arts and Entertainment
Jim Carrey and Jeff Daniels ride again in Dumb and Dumber To
filmReview: Dumb And Dumber To was a really stupid idea
Arts and Entertainment
Sir Ian McKellen tempts the Cookie Monster
tvSir Ian McKellen joins the Cookie Monster for a lesson on temptation
News
i100
Travel
Tourists bask in the sun beneath the skyscrapers of Dubai
travelBritish embassy uses social media campaign to issue travel advice for festive holiday-makers in UAE
Arts and Entertainment
Jennifer Saunders stars as Miss Windsor, Dennis's hysterical French teacher
filmJennifer Saunders and Kate Moss join David Walliams on set for TV adaptation of The Boy in the Dress
Life and Style
tech
Sport
Nabil Bentaleb (centre) celebrates putting Tottenham ahead
footballTottenham 4 Newcastle 0: Spurs fans dreaming of Wembley final after dominant win
Voices
Jimmy Mubenga died after being restrained on an aircraft by G4S escorts
voicesJonathan Cox: Tragedy of Jimmy Mubenga highlights lack of dignity shown to migrants
Life and Style
Sebastian Siemiatkowski is the 33-year-old co-founder and CEO of Klarna, which provides a simple way for people to buy things online
tech
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs General

Ashdown Group: Helpdesk Analyst

£25000 per annum: Ashdown Group: An established media firm based in Surrey is ...

Ashdown Group: Java Developer - Hertfordshire - £47,000 + bonus + benefits

£40000 - £470000 per annum + bonus: Ashdown Group: Java Developer / J2EE Devel...

Ashdown Group: Head of Finance - Financial Director - London - £70,000

£70000 per annum: Ashdown Group: Head of Finance - Financial Controller - Fina...

Recruitment Genius: Business Development Executive - Nationwide - OTE £65,000

£30000 - £65000 per annum: Recruitment Genius: This small technology business ...

Day In a Page

Jeb Bush vs Hillary Clinton: The power dynamics of the two first families

Jeb Bush vs Hillary Clinton

Karen Tumulty explores the power dynamics of the two first families
Stockholm is rivalling Silicon Valley with a hotbed of technology start-ups

Stockholm is rivalling Silicon Valley

The Swedish capital is home to two of the most popular video games in the world, as well as thousands of technology start-ups worth hundreds of millions of pounds – and it's all happened since 2009
Did Japanese workers really get their symbols mixed up and display Santa on a crucifix?

Crucified Santa: Urban myth refuses to die

The story goes that Japanese store workers created a life-size effigy of a smiling "Father Kurisumasu" attached to a facsimile of Our Lord's final instrument of torture
Jennifer Saunders and Kate Moss join David Walliams on set for TV adaptation of The Boy in the Dress

The Boy in the Dress: On set with the stars

Walliams' story about a boy who goes to school in a dress will be shown this Christmas
La Famille Bélier is being touted as this year's Amelie - so why are many in the deaf community outraged by it?

Deaf community outraged by La Famille Bélier

The new film tells the story of a deaf-mute farming family and is being touted as this year's Amelie
10 best high-end laptops

10 best high-end laptops

From lightweight and zippy devices to gaming beasts, we test the latest in top-spec portable computers
Michael Carberry: ‘After such a tough time, I’m not sure I will stay in the game’

Michael Carberry: ‘After such a tough time, I’m not sure I will stay in the game’

The batsman has grown disillusioned after England’s Ashes debacle and allegations linking him to the Pietersen affair
Susie Wolff: A driving force in battle for equality behind the wheel

Susie Wolff: A driving force in battle for equality behind the wheel

The Williams driver has had plenty of doubters, but hopes she will be judged by her ability in the cockpit
Adam Gemili interview: 'No abs Adam' plans to muscle in on Usain Bolt's turf

'No abs Adam' plans to muscle in on Usain Bolt's turf

After a year touched by tragedy, Adam Gemili wants to become the sixth Briton to run a sub-10sec 100m
Calls for a military mental health 'quality mark'

Homeless Veterans campaign

Expert calls for military mental health 'quality mark'
Racton Man: Analysis shows famous skeleton was a 6ft Bronze Age superman

Meet Racton Man

Analysis shows famous skeleton was a 6ft Bronze Age superman
Garden Bridge: St Paul’s adds to £175m project’s troubled waters

Garden Bridge

St Paul’s adds to £175m project’s troubled waters
Stuff your own Christmas mouse ornament: An evening class in taxidermy with a festive feel

Stuff your own Christmas mouse ornament

An evening class in taxidermy with a festive feel
Joint Enterprise: The legal doctrine which critics say has caused hundreds of miscarriages of justice

Joint Enterprise

The legal doctrine which critics say has caused hundreds of miscarriages of justice
Freud and Eros: Love, Lust and Longing at the Freud Museum: Objects of Desire

Freud and Eros

Love, Lust and Longing at the Freud Museum