Heroes of Zeroes: Nate Silver, his rivals and the big electoral data revolution
Statisticians' predictions have revolutionised political punditry and basketball alike. How did the big data revolution come about?
Tim Walker is The Independent’s Los Angeles correspondent, covering entertainment and other concerns from the West Coast of the US. He was previously a features writer and the editor of the paper’s diary column. His first novel, Completion, is being published in January 2014.
Friday 26 July 2013
If you listened to the pundits with the loudest mouths in the lead-up to last year’s US Presidential election, you probably thought it would be a close run thing.
If, on the other hand, you read Nate Silver’s blog, then you knew that President Obama had a healthy chance – a 90 per cent chance, in fact – of staying in the White House. Silver, a data journalist who at the 2008 election accurately forecasted the results in 49 states, called all 50 correctly in 2012.
This week, the value of Silver’s predictions became even clearer, when it was announced that the 35-year-old was leaving his berth at The New York Times and taking his blog, FiveThirtyEight, to the ESPN network, where he will oversee a new site devoted to statistical analyses of sports, politics and other fields.
Professor Marie Davidian, president of the American Statistical Association, says she noticed an “uptick” in attention in her subject in recent years, as the so-called “big data revolution” began. “But Nate Silver’s election predictions put a much broader spotlight on statistics. It resonated with people that someone could predict the election perfectly with dispassionate use of data.”
This is the International Year of Statistics, and as Silver’s ESPN deal demonstrates, there’s a growing market for people who can turn complex numbers into nuggets that the public can understand, and professionals can put to use – and not just for politics.
Following his exit, the New York Times public editor Margaret Sullivan revealed in a blog that Silver’s statistical approach to poll analysis had unsettled the pundit class. “A number of traditional and well-respected Times journalists disliked his work,” she wrote. Silver began his career as a baseball statistician, and the scepticism of his elders closely resembled that of the Oakland A’s scouts to the new-fangled strategy of general manager Billy Beane, in the movie Moneyball.
Beane’s method of picking players, which brought the As a record 20-game winning streak during the 2002 Major League Baseball season, was based on the “sabermetrics” system devised by statistician Bill James, whose work inspired people across the sporting world. In the early 2000s, for example, the basketball statistician John Hollinger, now vice president of the Memphis Grizzlies, created the player efficiency rating (PER), a formula to quantify the skills of any National Basketball Association player.
Ben Alamar, professor of sports management at California’s Menlo College, has consulted for several teams. If there’s a superstar analyst in American sports today, he says, it’s Dean Oliver, director of production analytics at, yes, ESPN. “Statistical analysis in basketball and football is a very new field,” Alamar says. “When I started around nine years ago, there was just one analyst working in the NBA. Now most teams have one.”
Oliver, who has worked with both the Seattle Supersonics and the Denver Nuggets, wrote the definitive book on basketball analysis, Basketball on Paper. “He was the first significant analyst employed by a team,” Alamar explains. “His role now is to develop metrics for every sport that ESPN is interested in.”
Dr Jeff Leek, who teaches at Johns Hopkins University’s school of public health in Baltimore, is the co-editor of the blog SimplyStatistics.org. “Everyone has become a statistician on some level,” he says. “Every article you read on ESPN has some statistical argument in it. The price of data has dramatically dropped, as has the price of software that makes it easy to analyse them. So there’s a bigger group of people with access to useful data.”
In Leek’s field of public health, Hans Rosling captured the imagination of millions with a pair of inspirational talks in 2006 and 2007. The 64-year-old Swedish doctor and his Gapminder foundation developed software which turns complicated figures for, say, global health or economics, into interactive graphics. “He’s like Nate Silver, in that he can distil down relatively complicated ideas into understandable, digestible bits,” says Leek. “I consider him a rock star in the field.”
Many trace the rise of statistics to a 2009 quote by Hal Varian, Google’s chief economist. “I keep saying that the sexy job in the next 10 years will be statisticians,” Varian told The New York Times. The subject is seeing a boom in college applications, while a 2011 study by the McKinsey Global Institute suggested the demand for workers with data analytic skills would rise to four million worldwide in the next five years.
Much of that demand is driven by tech companies like Varian’s own, which are both the creators and consumers of vast amounts of data. In 2009, Netflix awarded a $1m prize to a team of researchers, Bellkor’s Pragmatic Chaos, who won the competition to create an algorithm to predict user ratings for films. The team’s lead statistician was Chris Volinsky, director of the statistics department at AT&T, another stats rock star. Says Davidian, “There are rock stars, and then there are rock bands: statisticians frequently work in teams.”
One such rock band was the Obama 2012 campaign’s data analytics team, who not only predicted the President’s electoral success, but helped to make it happen. Its frontman was 30-year-old Dan Wagner. In the closing months of the 2012 race, Obama’s top advisers were troubled by the President’s fluctuating poll numbers in the key swing state of Ohio. Wagner’s statistical model, however, suggested Obama was comfortably, consistently ahead. Like Silver, he had seen the future.
sportLiverpool 5 Norwich City 1: Uruguayan striker has now scored 11 league goals against the club
arts + entsOlivier-nominated actor and singer is set to star in Lloyd Webber's musical about the Profumo affair
filmWith more than 70 per cent of early films lost, archivists are scouring the world to preserve the precious examples that remain
sportThe coach of Chalfont St Peter's under-10s football team was relieved of his duties after he sent an email to parents that said: 'I am only interested in winning'
techA piece of new hi-tech kit aims to get us scribbling again
indybestMake getting out of the wrong side of bed on cold winter mornings a thing of the past with our selection of night-time covers
life + styleClarissa Baldwin is the brains behind the slogan 'A Dog is for Life not just for Christmas'
Work until you’re 70: Chancellor accused of ‘living in fantasy land’ over Autumn Statement pension reforms
What made Charles Saatchi grab my throat, by Nigella Lawson - as she accuses him of threatening to 'destroy' her with drug claims
Is this the scariest advert ever? Japanese tyre commercial comes with its own disclaimer and health warning
‘Put it in my mouth’: Viewers outraged by apparent reference to oral sex in VIP e-cig advert
Paul Walker death: Eight-year-old son of Porsche driver Roger Rodas tried to rescue his father
- 1 North Korea: Kim Jong Un 'sacks powerful uncle and has his aides executed'
- 2 The hardwired difference between male and female brains could explain why men are 'better at map reading'
- 3 Is this the scariest advert ever? Japanese tyre commercial comes with its own disclaimer and health warning
- 4 A forgotten episode in Russian history leaves links with the Philippines
- 5 ‘Put it in my mouth’: Viewers outraged by apparent reference to oral sex in VIP e-cig advert
£50000 - £70000 per annum + London: Harrington Starr: Senior Automation QA Eng...
£35000 - £50000 per annum + Benefits : Harrington Starr: SQL 2008 R2/2012 Deve...
£38000 - £45000 per annum + Benefits: Pro-Recruitment Group: Creative Audit Se...
£45000 - £50000 per annum + Benefits: Harrington Starr: SQL Developer (TSQL, P...