Allen Lane £30 (1032pp) £27 (free p&p) from the Independent Bookshop: 08430 600 030

The Defence of the Realm, By Christopher Andrew

TThe fact that this is not so much "a" history of MI5 as "the authorized history" is underlined by its sombre black jacket, which gives it the appearance of a British government document. In a very real sense, it is. For although the book has been commercially published and Christopher Andrew is an academic historian, it was commissioned by MI5. Once he had been appointed to the post of official historian, the book was written on MI5 premises.

This is a centenary history, starting with the founding of the security service in 1909. In the following year, MI5 and the Secret Intelligence Service (MI6) were created as separate services. Andrew engagingly charts the evolution of MI5 through two world wars, the Cold War, and now the war against fundamentalist terrorists.

Inevitably, the agency's focus has shifted from counter-espionage and counter-subversion to counter-terrorism. What hasn't changed is its clandestine nature. The existence of MI5 was not acknowledged officially until 1979, when Mrs Thatcher unmasked Sir Anthony Blunt as the Fourth Man in a statement to the Commons.

MI5 has to tread a wire: it needs to be secret to protect parliamentary democracy, but is under pressure to be as transparent as possible. This has led to an uneasy compromise. Andrew was given unrestricted access to almost 400,000 files, but most are referenced only as "Security Service Archives" and closed to the rest of us.

This puts him in a difficult position: if he can't share his key sources, he must ask his readers to take his analysis on trust. We have only limited means of evaluating MI5's official view, as transmitted by Andrew, on various critical issues – such as the "Wilson plot" of the 1970s or the "Death on the Rock" episode in Gibraltar in 1988.

Publicity material for the book has filled newspapers with stories of double agents and courageous derring-do. But what also emerges from these pages is a fascinating picture of MI5 as a tightly-knit institution, thriving on its atmosphere of secrecy. "No one, not even our own families, should be told where we worked or for whom," was the firm instruction to a new employee in 1931. Morale sank after the end of the Cold War and the Good Friday agreement of 1998, which led to massive cutbacks. But after 9/11, there was a rapid expansion of staff and a renewed sense of purpose.

The Service had a choral society, which takes its name "The Oberon Singers" from Oberon's words in A Midsummer Night's Dream – "We are invisible, we will o'er hear their conference." It also had a cricket team and cricket imagery frequently appears in correspondence. "So the first XI of MI5 is to play the Mau Mau," commented the head of the Overseas Division when MI5 officers were sent to Kenya in 1952.

The Service did not advertise openly for recruits before 1997: until then, recruitment was based on personal recommendation. This was a narrow social group, many of whom had served in India or elsewhere in the Empire.

Male officers listed their recreations as cricket and hunting, while women were graduates of elite schools and universities. Women have always played an important role in MI5 and two recent Director Generals have been female – Stella Rimington, one of the first women agent-runners, and Eliza Manningham-Buller.

Right up to the mid-1970s, the post-war Service refused to recruit Jews on the grounds that a dual loyalty to both Britain and Israel might create a conflict of interest. This was "inexcusable", Andrew rightly observes. So too was the attitude to black people of Guy Liddell, Deputy Director General. "It was true," he told the Joint Intelligence Committee in 1949, "that niggers coming here often went to the C[ommunist] P[arty]." There was no doubt in his mind that "West African natives are wholly unfitted for self-rule."

Shockingly, the Service carried out secret surveillance of the colonial delegations which came to London to discuss terms for independence in the 1950s and 1960s. Andrew gives a disturbing account of the stealthy gathering of intelligence on the delegates attending conferences which negotiated the independence of Cyprus and Kenya. The Home Secretary, Rab Butler, cynically condoned these operations on the grounds that "obviously the product was of great importance and of great value to the government negotiators".

In most of the Empire, claims Andrew, MI5 contributed to a smooth transfer of power through the work of its liaison officers. But Guyana, where Churchill wanted to "break the Communist teeth", was a shameful exception. Here, MI5 supported British and American covert action to oust the democratically-elected Cheddi Jagan from power. Andrew claims that the Service was not "directly" involved and that the dominant intelligence agency in the years leading up to independence in 1966 was the CIA. But neither point exculpates MI5 or, more pertinently, the British government.

The Service has had some remarkable achievements, notably the Double-Cross System of the Second World War, which fed disinformation to the Germans. For the most part, however, it is difficult to measure MI5's success, since it can only be judged by things which do not happen – like the prevention of sabotage.

But it seems astonishing that it was not until 1951, as the result of the decrypt of a KGB telegram, that any of the Cambridge Five – all MI5 or MI6 employees, recruited at Cambridge in the 1930s to become spies for the Soviet Union – were identified. The decrypt took the Service completely by surprise and began the most drawn-out investigation in its history, taking over 30 years to complete.

The thousand pages of this book are brimming with some wonderful details. But many could be pruned – like the fact that the first Director's garden contained "400 rose trees and a grass tennis court". This would make room, perhaps, for a fuller account of the human factor in spying.

What was the intellectual and psychological motivation of the staff of MI5? – not only of the spies, agent-runners, and codebreakers, but those who steamed open envelopes and eavesdropped on telephone conversations. How did they feel about deceit?

The Defence of the Realm is a valuable and important contribution to our understanding of the 20th century. But an official history can only do so much, especially of an organisation that is inherently secret. In this hazy world of smoke, mirrors and lies – where actual conspiracies are barely distinguishable from conspiracy theories – we also need the scrutiny of genuinely independent investigators, such as Robin Ramsay, the maverick editor of the journal Lobster, and of unofficial historians. It will be interesting to compare The Defence of the Realm with the authorised history of MI6, which is set to follow next year.

Susan Williams is a Senior Fellow at the Institute of Commonwealth Studies, University of London. Her latest book is 'Colour Bar' (Penguin)

Independent Comment
blog comments powered by Disqus

Day In a Page

Picture preview: Lucian Freud drawings

Lucian Freud drawings

Picture preview
Silent revolution at the Baftas as the French take top awards

Silent revolution at the Baftas

The Artist wins in seven categories, with Meryl Streep the other big success story
Whitney Houston: The diva who had – and lost – it all

The diva who had – and lost – it all

Nick Hasted charts the highs and lows of Whitney Houston's life
How Picasso won over (some of) the British

How Picasso won over (some of) the British

Winston Churchill and Evelyn Waugh hated his work, but Picasso provided inspiration for a whole generation of UK artists
Topshop: A Decade Of Design

Topshop: A Decade Of Design

When London Fashion Week starts on Friday, Topshop will celebrate 10 years backing its brightest young stars
John Prescott: 'My wife thought I'd just retire, but I'm not a slippers man'

'My wife thought I'd just retire, but I'm not a slippers man'

At 73, John Prescott isn't mellowing. In fact he's taking a shot at becoming a police commissioner
Jim Gamble: We are losing the race to protect our young

Jim Gamble: We are losing the race to protect our young

Technology and the children who use it won't wait for slow-moving child-protection services and police to catch up
Sarah Sands: A friend is not the one you turn to, but the person who turns to you

Sarah Sands on friendship

A friend is not the one you turn to, but the person who turns to you
Andy Burnham: 'It's a genie out of the bottle moment'

Andy Burnham interview

'It's a genie out of the bottle moment'
Leveson: What we've learnt so far

Leveson: What we've learnt so far

Ingenious hacks, shifty editors and attacks of Sudden Memory Loss Syndrome – Matthew Bell assesses the state of play at the Royal Courts of Justice
Modern lovers: The 'sexual body warriors' and pioneers transforming 21st-century relationships

Modern lovers: The 'sexual body warriors'

Sarah Morrison meets the people redefining love in the 21st century.
'I was angry, so angry': How heartbreak, betrayal and Su Pollard helped Estelle find pop success

Estelle: 'I was angry, so angry'

The singer talks about heartache, betrayal and bouncing back.
Choc tactics: Bill Granger's Valentine's recipes for chocoholics

Bill Granger's Valentine's recipes for chocoholics

Should it be white, milk or plain? Can you make a melt-in-the-mouth pudding without using any?
Male, pale & stale: Could more women on the board help Mothercare – and other ailing firms?

Male, pale & stale

Could more women on the board help Mothercare – and other ailing firms?
Upstairs, downstairs, 2012-style

Upstairs, downstairs, 2012-style

There are now more domestic workers in Britain than in Edwardian times