Revealed at last: the abdication secrets buried for 66 years

Documents deemed too sensitive to publish while Queen Mother lived reveal bizarre antics at palace - and a secret lover

The black limousine glided to a halt shortly before 9pm in a dark courtyard at the rear of Buckingham Palace. Unceremoniously, the car's passenger was lifted through a window by royal footmen.

The clandestine visitor, climbing like a burglar into the Royal Family's private quarters on 3 December 1936, was the Right Hon Stanley Baldwin MP, the Prime Minister.

Downing Street had been called at 6pm. King Edward VIII needed to see Baldwin in the utmost secrecy. The monarch was making one last desperate attempt to keep his throne and marry Wallis Warfield Simpson – the American divorcee whose presence at his side was scandalising British society. The attempt failed. He abdicated eight days later.

Secret government papers released today at the Public Record Office in Kew reveal a web of intrigue and deceit by state and monarchy that has remained hidden for 66 years.

It ranges from Special Branch surveillance of Mrs Simpson's sex life to the direct intervention of the late Queen Mother in ensuring Edward and his wife would be frozen out of royal circles for life.

The abdication papers, which were kept from public view until the Queen Mother's death last year, show that Edward VIII tried to outmanoeuvre the Government and hold on to his crown with a direct appeal to the British people.

He had summoned Baldwin to tell him of his intention to make a broadcast on the BBC the next day in which he would declare his love for Mrs Simpson and plead that she be accepted as his wife.

Documents giving details of the conversation between the Prime Minister and the King show the two men became implacably opposed over the issue, which resolved the Cabinet to force the abdication.

The text of the speech, the existence of which was unknown outside Whitehall until today, was a carefully tailored, unashamed appeal to popular sentiment. It was written with the help of a friend of the King and Conservative MP whose reputation for rhetoric was soon to become unsurpassed – Winston Churchill.

Noting that "by ancient custom" a ruler speaks directly to his subjects, Edward said he could no longer be King without marrying. He wrote: "It has taken me a long time to find the woman I want to make my wife. Without her, I have been a very lonely man. With her I shall have a home and all the companionship and mutual sympathy and understanding which married life can bring."

Adding that neither he nor Mrs Simpson, the wife of a retired Army officer whom he had met four years earlier and started a relationship with in 1934, expected that she would become Queen. All he wanted was a "proper title and dignity for her, befitting my wife".

The speech ended with Edward saying he would now go away for an unspecified period "so that you may reflect calmly and quietly".

The reaction in Whitehall was anything but, according to the documents. A hurriedly scribbled note in pencil at the bottom of the address sums up the reason for the disquiet: "No word about abdication."

Baldwin, the Conservative leader of the all-party National Government, had already made clear the opposition of not only the Cabinet but also Clement Attlee's Labour Party to the King's proposal of a "morganatic" marriage under which Mrs Simpson would have none of the status of a royal consort. In one Downing Street memo, the Prime Minister said any attempt by Edward to remain King was likely to cause "the gravest injury to national and imperial interests".

The Cabinet resolved to block the King's proposed speech on a constitutional ground, saying that any public statement from the monarch could only be made with the approval of ministers.

A report on the address, hurriedly written on 4 December, said that if Edward went ahead "constitutional monarchy would cease to exist". It added: "The constitutional duty of the King is to take no public action which is calculated to divide his subjects into opposing camps. It is manifest that the King's broadcast must have this effect."

Letters received from the dominions, the self-governing areas belonging to the British Empire, also revealed growing opposition, led by Australia and Canada. Only the Prime Minister of New Zealand wrote to express support for Edward remaining on the throne.

Underpinning the hardline position of the Cabinet was also the fear that Churchill, who had written to Baldwin saying it would be a "cruel thing" to pressure Edward into coming to a rapid decision, would form a breakaway political party in support of the King.

When told of the Cabinet's opposition to his broadcast, it seems Edward reluctantly accepted the impossibility of his position. At the end of the discussion at Buckingham Palace, during which Baldwin stalled for time by telling Edward that his proposed broadcast was unconstitutional, the King said: "You want me to go, don't you?"

A further document recording another meeting between Baldwin and Edward at the King's Windsor residence, Fort Belvedere, noted: "The King said that before long he would have to abdicate if he married."

The documents, which were initially due to be released after 30 years but were made subject to a 100-year restriction lifted by the Government last year, offer few further insights into longstanding rumours about the main players in the abdication.

Those hoping that the papers would finally lift the lid on suspicions that the couple, who became the Duke and Duchess of Windsor, were Nazi sympathisers will be disappointed. The sole mention is a note in 1936 to the Chancellor of the Exchequer, Neville Chamberlain, that Wallis had tried to contact the German regime and had mentioned "doing a flit" to Germany in 1937.

But the dislike of the Royal Family's senior females, including the Queen Mother, for the Duchess is made clear in a 1938 letter from the new King, George VI, to Chamberlain – by then Prime Minister – about proposals for his brother to visit Britain. George wrote: "I think you know that neither the Queen or Queen Mary [his mother] have any desire to meet the Duchess of Windsor."

The split was so final that George could not even bring himself to reveal it to Edward. He told Chamberlain: "Perhaps my brother would take this decision in a more friendly manner from you than me."

Speech the King was not allowed to give

Extracts from a proposed address by Edward VIII to save his crown.

"I realise that the newspapers of other countries have given you full cause for speculation as to what I am going to do.

It was never my intention to hide anything from you. Hitherto it has not been possible for me to speak, but now I must.

I could not go on bearing the heavy burdens that constantly rest on me as King unless I could be strengthened in the task by a happy married life; and so I am firmly resolved to marry the woman I love, when she is free to marry to me.

You know me well enough to understand that I never could have contemplated a marriage of convenience. It has taken me a long time to find the woman I want to make my wife. Without her I have been a very lonely man. With her I shall have a home and all the companionship and mutual sympathy and understanding which married life can bring. Neither Mrs Simpson nor I have ever sought to insist that she should be Queen. All we desired was that our married happiness should carry with it a proper title and dignity for her, befitting my wife."