Although he reluctantly decided to step down last night, the 50-year- old father-of-three was defiant about his private life, the details of which had been splashed across the Sunday papers in conventionally prurient style.
Yet even by liberal Labour standards, let alone the strict moral sanctions of the Tory party, Mr Spencer's lifestyle was determinedly unconventional.
His wife, Liz, had known he was gay ever since the couple first met as students 26 years ago, and accepted that he would frequently make trips to Amsterdam and other places to indulge his pleasures.
"Like the song from Les Miserables, `There are some things that are better left unspoken, better left unsaid'," she said. "We have always promised to be completely honest with each other and Thomas has always told me everything that I have asked him."
Mrs Spencer made clear that she respected his double life and did not ask about his trips away or his sexual partners or the videos he watched.
However, even she was unaware of Mr Spencer's drug use, and the discovery of not just cannabis joints but cocaine was the main factor in the affair which was to ensure that he could no longer stand for the Tories at the European elections in June.
Mr Spencer had anyway become worried about the Tories' increasing Euroscepticism, and it was noticeable that Kenneth Clarke, the former Tory Chancellor, was a lone voice yesterday within the party to offer his support to the beleaguered MEP.
Last night Mr Spencer said: "At 5.45pm I spoke to Michael Ancram, the chairman of the Conservative Party, who has been kind enough to pay tribute to the work I have done for the party.
"I have concluded it is in the best interests of all concerned that I should withdraw from the party's list for the next European elections."
The incident rounded off a disastrous week for the Tories, coming in the wake of reports of storms within the shadow cabinet and polls that gave the party their lowest rating since the election.
However, Conservative Central Office was last night trying to put a positive spin on the latest scandal sure to affect the party's already low popularity.
"This is a vindication of the system that William Hague has set up," one Tory source said. "In the past, the Tory party didn't have a centralised disciplinary structure and that's why you had the spectacle of people like [Neil] Hamilton dithering over resignation. Spencer saw which way the wind was blowing and went."
Last night Mr Spencer bowed out with the dignity that he and his wife had attempted to maintain throughout the most turbulent 24 hours of their lives.
"I would like to thank all the journalists whose individual courtesy has made the last two days bearable," he said.