The (very) secret history of Area 51

Just how covert is the infamous US air base? New files show that even Presidents don't always 'need to know' its activities

No one on the ground spotted Area 51's latest toy as it kept watch on Osama bin Laden's compound in Abbottabad, Pakistan, on the night of the US raid.

Rather than one of the UFOs that the wilder fringes of the internet believe the military has stashed away at America's top-secret military site in Nevada, this "toy" was actually the latest unmanned aerial vehicle (UAV), developed at the secretive base. Named the "Beast of Kandahar" after it was snapped at Kandahar air base in Afghanistan back in 2009, this stealthy grey batwing-shaped reconnaissance drone, officially known as the RQ-170 Sentinel, was a throwback to Area 51's golden age before the advent of the spy satellite put the spy plane out of business.

While Area 51 will always be associated with conspiracy theories, for journalists such as Annie Jacobsen its purpose was – and is – to develop and test the latest kit from the military industrial complex that maintains America's superpower status.

"Area 51 was the single most important Cold War facility as it was set up to push science faster and further than the Soviet Union," says Jacobsen, author of Area 51: An Uncensored History. "On one side of the road the Nevada test site was preparing for the Third World War, and on the Area 51 side they were trying to prevent it." Yet for Jacobsen one of the strangest things about Area 51 is that she has "not seen anything ever leak out of Area 51", almost as if "the base is in a permanent state of lockdown".

Sitting on the edge of a dried-up lake bed cradled by mountains, only 90 miles or so from Las Vegas, it is perhaps not surprising that it is hard to separate the myth from the fact of Area 51, or Groom Lake, as old-timers have traditionally called it.

Recent declassified documents shed light on the facts behind the myths of Area 51, from the U-2 spy plane missions that helped to unlock the secrets of the Soviet Union in the 1950s to the groundbreaking stealthy A12 that was obsolete before it even first served its country, and the record-breakingly fast recon plane the SR-71 Blackbird, which helped to spot North Vietnamese missile bases in the late 1960s and 1970s; and from the beginnings of stealth technology to the development of the F-117 stealth fighter itself. Also brought to light was the test flying of "acquired" Soviet MIG fighters in mock combat situations, which led to the founding of the Top Gun pilot programme.

The isolation of the base may be one good reason why the Beast of Kandahar was discovered on the opposite side of the world and not in Nevada. Along with the ability to time testing activities for when spy satellites have already passed the base, it also allows for the capacity to "go underground" to avoid prying eyes. Yet for Jacobsen it is the "need to know" principles of the special-access "black" programmes that run out of the base that account for the cloak of secrecy that has been maintained. These principles were developed from the Manhattan Project, which saw the creation of the world's first nuclear bomb during the Second World War.

Sometimes even the president is kept in the dark. According to Jacobsen, during the 1994 inquiry into allegations of Cold War human radiation experiments, "certain records involving programmes at Area 51 were kept from President Clinton because he didn't have a need to know".

For ex-Area 51 radar man Thornton D Barnes, the need to know meant "everything was compartmentalised", with only a small number of highly screened workers allowed to remain permanently on the base. Everyone else had to fly in from Vegas to "prevent fresh faces showing up in the local communities and drawing attention to something that might be going on".

"Even now we have retired, we don't ask each other questions about what we did unless we know that it has been declassified."

According to Bill Sweetman, editor-in-chief of defence technology for Aviation Week, simply "pulling out the plug" and "supergluing USB ports" has also helped to keep these black programmes disconnected from our interconnected world. So successful are the secrecy techniques that it's clear that we are going to see more – not fewer – of these black programmes. However, Sweetman believes that – along with strict secrecy – "black propaganda" helps to keep Area 51's secrets.

"There were certainly deliberate disinformation campaigns to generate a lot of noise about UFOs back in the 1950s and 1960s to cover secret flights of planes like the U-2, and then again in the late 1970s and early 1980s to link Area 51 to UFOs through 'fake' documents and eyewitness accounts of alien technology – and even alien bodies." The goal, he believes, was to mask the rebuilding of the base and then to stop people "asking why is there this huge secret and expensive military base in the middle of the desert".

Annie Jacobsen has real fears about the "lack of respect for the Constitution" that this level of secrecy can entail. The construction in 2007 of a new hangar twice the size of any other at Area 51 and another large hangar in 2010 meant that something was going on – but the public doesn't have a need to know.

It is possible to look at the Beast of Kandahar for clues as to what the new project might be, or even at the Special Forces Black Hawk helicopters that were also revealed or even compromised by the Bin Laden raid. The size of the hangars suggests it could be early prototypes of the $4bn next-generation American bomber programme that are already being put through their paces down on Groom Lake.

Whatever it is, the Beast of Kandahar is only the shape of things to come.