'We try to collect everything and hang on to it forever': US intelligence agencies' cosy relationship with academia and business may be hard to unwind

8 June 2013: US President Barack Obama, right, and Chinese President Xi Jinping at the Annenberg Retreat at Sunnylands in Rancho Mirage, California where they were finishing their summit

Revelations of snooping into private data and communications on a massive scale by the Obama administration has shed a little daylight on the widespread and lucrative links between American intelligence agencies, industry and academia.

The Pentagon has long funded research into new technology, to the extent that it is claimed that Silicon Valley would not have developed without this revenue stream. The CIA formed its own non-profit private enterprise, In-Q-Tel, in 1999, specialising in advancing the development of technology for collecting and analysing information, which continues to back nearly 60 companies to this day.

The practice of the US government using business soared after the 9/11 attacks, with thousands like the whistleblower Edward Snowden recruited for George W. Bush’s War on Terror. The Office of the Director of National Intelligence (ODNI) in Washington estimated recently that almost one in four people working in the espionage field were in the pay of private concerns and no less than 70 per cent of the intelligence budget went to companies like Mr Snowden’s employer, Booz Allen Hamilton.

An ODNI briefing paper stated that the philosophy had become: “We Can’t Spy … If We Can’t Buy!” By 2010 about 1,930 private companies worked on programs related to counterterrorism, homeland security and intelligence at more than 10,000 sites across the country.

Mr Snowden’s first job linked with the National Security Agency (NSA) was at Maryland University, where he worked as a security guard at a facility run by the agency.

Academic institutions are natural laboratories for developing cutting-edge technology across the developed world. In Britain GCHQ, the Government’s intelligence communications centre, is funding research into cyber threats at Oxford, Bristol, Queen’s University Belfast, Southampton, and Imperial, University College and Royal Holloway in London. These “centres of excellence”, GCHQ says, “will help the UK’s cyber knowledge through original research; provide top quality graduates for the field, support GCHQ’s cyber defence mission and drive up the level of innovation”.

The thirst for information is seemingly insatiable among some in the intelligence community. Ira Hunt, the CIA’s chief technology officer, told a conference in New York three months ago: “More is always better… Since you can’t connect the dots you don’t have, we fundamentally try to collect everything and hang on to it forever. It is really very nearly within our grasp to be able to compute on all human-generated information.”

Mr Snowden was one of those brought in to “connect the dots” as the tempo of intelligence gathering using new technology has continued to grow. They work side by side with civil servants as analysts, technical support specialists and mission managers, often with access to secret and top-secret material.

Stewart Baker, a former assistant secretary at the Department of Homeland Security, said the Snowden affair might lead to changes. “I have no doubt this is going to produce some soul-searching about how many contractors the intelligence community has and what they have access to, and how much vetting is done on the employees,” he said.

But the relations within the security-industrial complex may be too enmeshed and cosy to change . Booz Allen Hamilton’s clients include the NSA, CIA and ODNI. Its executive vice-president and head of national security, Mike McConnell, is a former director of the NSA.