Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution

Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?

If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.

If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.

This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.

Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”

This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.

Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.

Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.

Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”

Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”

Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”

In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”

Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.

Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”

One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.

PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
News
ebooksAn unforgettable anthology of contemporary reportage
Life and Style
tech

Board creates magnetic field to achieve lift

News
There have been various incidents of social media users inadvertently flouting the law
news

Life and Style
Stack ‘em high?: quantity doesn’t always trump quality, as Friends of the Earth can testify
techThe proliferation of online petitions allows us to register our protests at the touch of a button. But do they change anything?
News
Bourgogne wine maker Laboure-Roi vice president Thibault Garin (L) offers the company's 2013 Beaujolais Nouveau wine to the guest in the wine spa at the Hakone Yunessun spa resort facilities in Hakone town, Kanagawa prefecture, some 100-kilometre west of Tokyo
i100
Sport
CSKA Moscow celebrate after equalising with a late penalty
footballCSKA Moscow 2 Manchester City 2: Premier League champions let two goal lead slip in Russia
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs General

IT Project Manager

Competitive: Clearwater People Solutions Ltd: Our client based in Chelmsford a...

Business Intelligence Specialist - work from home

£40000 per annum: Ashdown Group: An established and growing IT Consultancy fir...

Business Intelligence Specialist - work from home

£40000 per annum: Ashdown Group: An established and growing IT Consultancy fir...

IT Manager

£40000 - £45000 per annum + pension, healthcare,25 days: Ashdown Group: An est...

Day In a Page

Indiana serial killer? Man arrested for murdering teenage prostitute confesses to six other murders - and police fear there could be many more

A new American serial killer?

Police fear man arrested for murder of teen prostitute could be responsible for killing spree dating back 20 years
Sweetie, the fake 10-year-old girl designed to catch online predators, claims her first scalp

Sting to trap paedophiles may not carry weight in UK courts

Computer image of ‘Sweetie’ represented entrapment, experts say
Fukushima nuclear crisis: Evacuees still stuck in cramped emergency housing three years on - and may never return home

Return to Fukushima – a land they will never call home again

Evacuees still stuck in cramped emergency housing three years on from nuclear disaster
Wildlife Photographer of the Year: Intimate image of resting lions claims top prize

Wildlife Photographer of the Year

Intimate image of resting lions claims top prize
Online petitions: Sign here to change the world

Want to change the world? Just sign here

The proliferation of online petitions allows us to register our protests at the touch of a button. But do they change anything?
Ed Sheeran hits back after being labelled too boring to headline festivals

'You need me, I don’t need you'

Ed Sheeran hits back after being labelled too boring to headline festivals
How to Get Away with Murder: Shonda Rhimes reinvents the legal drama

How to Get Away with Murder

Shonda Rhimes reinvents the legal drama
A cup of tea is every worker's right

Hard to swallow

Three hospitals in Leicester have banned their staff from drinking tea and coffee in public areas. Christopher Hirst explains why he thinks that a cuppa is every worker's right
Which animals are nearly extinct?

Which animals are nearly extinct?

Conservationists in Kenya are in mourning after the death of a white northern rhino, which has left the species with a single male. These are the other species on the brink
12 best children's shoes

Perfect for leaf-kicking: 12 best children's shoes

Find footwear perfect to keep kids' feet protected this autumn
Anderlecht vs Arsenal: Gunners' ray of light Aaron Ramsey shines again

Arsenal’s ray of light ready to shine again

Aaron Ramsey’s injury record has prompted a club investigation. For now, the midfielder is just happy to be fit to face Anderlecht in the Champions League
Comment: David Moyes' show of sensitivity thrown back in his face by former Manchester United manager Sir Alex Ferguson

Moyes’ show of sensitivity thrown back in his face... by Ferguson

Manchester United legend tramples on successor who resisted criticising his inheritance
Two super-sized ships have cruised into British waters, but how big can these behemoths get?

Super-sized ships: How big can they get?

Two of the largest vessels in the world cruised into UK waters last week
British doctors on brink of 'cure' for paralysis with spinal cord treatment

British doctors on brink of cure for paralysis

Sufferers can now be offered the possibility of cure thanks to a revolutionary implant of regenerative cells
Ranked seventh in world’s best tourist cities - not London, or Edinburgh, but Salisbury

Lonely Planet’s Best in Travel 2015

UK city beats Vienna, Paris and New York to be ranked seventh in world’s best tourist destinations - but it's not London