A "lost" hoard of emails sent by senior executives in Rupert Murdoch's newspaper empire at the height of the phone-hacking scandal has been found, The Independent has learnt.
Detectives from the Metropolitan Police are now expected to examine the database of emails in their renewed search for News of the World journalists who may have hacked into mobile phone messages or hired private detectives to do so in breach of privacy laws.
A senior editor at News International, the UK newspaper arm of Mr Murdoch's News Corporation media empire, claimed in a high-profile criminal trial last year that "lots of emails" from editors and other staff had gone missing in a botched data transfer to India.
However, The Independent has established that not only is the database intact but it apparently contains a full record of email traffic between the company's senior staff.
The archive, covering 2005 and 2006 when the News of the World was illegally listening to the messages of aides to Prince William and other public figures, will open a window onto the Sunday paper's newsgathering operations and those of Mr Murdoch's other titles.
Scotland Yard last week promised it would leave "no stone unturned" in its new inquiry into allegations that phone hacking at the News of the World spread far beyond its Royal Editor, Clive Goodman, whose snooping on Prince William led to him being jailed for four months in 2007. A raid on the offices of the private eye Glen Mulcaire, also jailed, uncovered several thousand phone numbers of potential hacking victims and 91 PIN codes, but detectives limited their inquiries to Goodman and Mulcaire.
Confirmation that the UK database of all emails does, in fact, exist will give the new police team no excuses for ignoring a data trail that may yield fresh clues to the investigation.
Saying it was determined to root out wrongdoing, News International last week passed detectives significant new evidence about the NOTW's sacked head of news Ian Edmondson, prompting the Met to launch a new inquiry. It is being run by Deputy Assistant Commissioner Sue Akers.
The claim that emails had been lost was made by the News of the World's Scotland editor Bob Bird during the perjury trial of the former Scottish Socialist Party leader Tommy Sheridan in November. Giving evidence on oath, he said that many emails sought by the defence had gone missing while being transferred to India.
The claim prompted the Information Commissioner's Office to launch an inquiry into whether the apparent loss of the emails broke the law governing the handling of sensitive data and its transfer abroad. If News International had broken the Data Protection Act it would have faced a fresh civil case, alongside civil actions brought by individuals who suspect their phones were hacked, and a £500,000 fine.
However, in a letter to the Information Commissioner's Office, lawyers at News International's Wapping headquarters told the ICO's investigation team that it had archived emails and that none had been transferred to India.
As well as confirming the presence of a potentially vast data store, the disclosure – confirmed by the newspaper group to The Independent – has other potentially serious consequences.
Mr Sheridan's team is expected to use the disclosure as part of an appeal against Mr Sheridan's conviction for lying on oath in his earlier defamation case against the NOTW which led to him being jailed for three years.
Secondly, if, as expected the ICO confirms News International is telling the truth, it could forward the transcript of Mr Bird's testimony and its inquiry to its lawyers who will decide whether "in the public interest" to contact other prosecuting authorities.
A source at News International said Mr Bird had unintentionally given the court inaccurate evidence, but insisted the defence team had received all the relevant documents. In a brief official statement, the news group said: "Like many companies, we have an email archiving system in place." Mr Bird, who joined News International in 2000, did not respond to a request for comment.
Aamer Anwar, Sheridan's solicitor, said: "This is unacceptable. We were told repeatedly by Mr Bird in these proceedings that this material was lost in Mumbai. Now we are told it is in the UK. My client would not accept an explanation that there was a misunderstanding. We will look closely at any response from News International and are considering a complaint to the police and the Crown Office. If there is evidence information was intentionally not supplied we would expect criminal proceedings."
Labour MP Tom Watson, who has campaigned for a new police force to look into hacking, said: "If the jury in the Sheridan trial was misled then there should be an urgent review of the case. This week the Prime Minister told the Commons that the inquiry should 'follow the evidence wherever it leads'. We now know it leads to a data warehouse in London containing all News International emails from 2005 onwards."
* The digital age has brought with it an unforeseen dilemma for commercial companies, from banks to media giants: just what do they do with the unimaginably vast quantities of data that they produce and receive each year? According to one estimate, 35 zettabytes of information – equivalent to the storage capacity of 2,625 billion iPads – will be generated worldwide in 2011.
Commercial lawyers say this avalanche of data presents the commercial sector with particular problems because of the thicket of legislation – from the Data Protection Act to rules governing the financial sector – that means nearly all of it must be kept and archived in an accessible manner.
The dangers of failing to properly store information such as internal emails was highlighted in 2009 when Barclays Bank was heavily criticised for allowing key documents in a dispute with a customer, including emails, to be destroyed. Although the banking giant ultimately won the case, the judge ordered the amount of costs it could claim to be halved.
Companies have traditionally relied on "taping" technology or hard discs, often housed in basement computer rooms, to back up their data. But increasingly companies are moving to online storage, which allows information to be transmitted in vast quantities via the internet to remote servers around the world.
Tracey Stretton, legal consultant for Kroll Ontrack, a data management and recovery company, said: "It is increasingly vital that companies not only store the information they generate, but that they do so in a manner that allows that data to be searched effectively should the need arise.
"If, for example, legal proceedings arise a number of years after an event, then there will be a reasonable expectation that any relevant information should be readily accessible.
"The quantity of information that technology allows to be stored is extraordinary – a single USB data stick can hold the equivalent of 20 tonnes of printed paper."
According to one estimate, the electronically stored information (ESI) industry is now worth £150bn a year.
Cahal Milmo and Martin Hickman
Mystery of missing emails
* What Bob Bird, editor of the Scottish News of the World, told a pre-trial hearing on 30 June 2010
"I did have a look at the email system but it is, frankly, a mess, our system. Our archived emails have been shipped to Mumbai and it's difficult to get anything that is more than six months old. I searched out emails regarding the police, which I'm still trying to discover where they actually are and how you open the things. I've had IT at it for a few weeks now..."
"... As I say, unfortunately, our emails get automatically deleted and archived after about six months now and our archive is in Mumbai, so I couldn't have searched [those] emails..."
"...I've been having a look trying to find anything that might be relevant. As I say, we've had a problem..."
During the trial in November, he told the High Court that "many emails had been lost when they were being moved to an archive in India".
* What News International subsequently told the Information Commissioner's data inquiry
As a result of Mr Bird's comments, the Information Commissioner's Office launched an investigation into whether News International's apparent archiving of emails abroad and their supposed loss breached the Data Protection Act. The ICO contacted News International and asked it to explain its position. Lawyers for News International then wrote to the headquarters of the Information Commissioner's Office in Wilmslow, Cheshire, stating that it archived emails in the UK and had not sent any to India.
News International has confirmed to The Independent that this is the case.