This week, in William Hague's time of trouble, they have come into the minds of many Tories. And, with this paper's jolly tale of Mr and Mrs Patten holidaying with the Majors at the home of the Conservative arch- plotter Tristan Garel-Jones in Spain to titillate them, he is much in their thoughts. The question is whether the Tory party and its leadership are in his.
One who observed him closely in Hong Kong said he was "completely exhausted" when he left and "pretty fed up". His parting words in Hong Kong were "I'm going to France - to write a book, to tend to my garden and to recharge my batteries, which are starting to splutter damply like an old car on a cold wet morning".
And that is exactly what he has been doing. With Lavender, his wife of 26 years, he retired to the beautiful old stone farmhouse they bought two years ago at St Martin-Languepie near Albi, the birthplace of Toulouse- Lautrec, in the south.
When a month or so later, a journalist negotiated the wildnerness near the hideaway to grab a word, Lavender politely and patiently explained that her husband was doing the ordinary job of shopping at the supermarket. "He just wants to be left alone," she said. And so he has been. He has laid flagstones to create a garden dining patio, planted fruit trees and dug up nettles. Undoubtedly, as a keen tennis player - and anxious weight- watcher - he will have been making the most of his new court. He is thought to be hesitant about a return to politics and Lavender, a barrister who had her own successful family practice before they left, is also thought to have her reservations. Yet he did leave the prospect open when he turned down a peerage in Major's resignation honours.
And he has been writing his book on Hong Kong and Asia for which the publishers HarperCollins paid a rumoured pounds 100,000 advance.
Whatever his wing of the Conservative Party may wish, he is not a man who needs to hurry. His writing could take at least until the end of the year. He was paid pounds 238,000 in his final years in Hong Kong with pounds 55,000 a year expenses. He left office with a pounds 275,000 tax-free gratuity and, it might be assumed, is not desperate for a new job yet.
But at some point he and Lavender will return to Britain, not to the converted Methodist chapel which was their home in Bath when that was his constituency but to London. After much hunting, they bought an attractive Victorian three-storey six-bedroom house with a large garden in Barnes, a desirable part of south-west London with easy access to Westminster. His youngest daughter, Alice, 17, is due to begin reading languages at Cambridge University. His elder daughters, Kate, 24, and Laura, 22, both work in the media, Kate in television for Esther Rantzen, Laura as a beauty assistant for the glossy magazine Harpers and Queen.
Will they find themselves the daughters of a man who has set his cap at the Conservative Party in the hope of becoming Prime Minister? "He's quite fastidious about not wanting to be parachuted back in with a by- election," said one insider. Though the seats of John Major and Michael Heseltine have been mentioned as possibilities, "He wouldn't want anyone to stand down for him."
And the Conservative Party, at this moment, riven by conflict and facing a government with a massive majority, is not the most attractive proposition for a man who has other options.
"The thing about Patten," says one who knows him well, "is he does believe that big jobs are important. He might well want to be Prime Minister but in a choice between waiting in opposition for three terms and having something like a big job in Europe, there's no doubt which he would prefer." His name, like Kenneth Clarke's, has been mooted as a potential successor to Jacques Santer, though he would have to become a European commissioner first.
Patten gave his all to his post in Hong Kong and the observer thinks Patten enjoyed the international stage. "If he had his way, he would want a big international job - he thought overseas development minister was the best job he had at Westminster. But he knows the big ones are largely in the gift of the government of the day and he's not flavour of the month with them."
Patten must have thought when he went to Hong Kong that he was leaving his political career behind him, but the Tory world he left was very different from the one he would return to. His friend John Major was Prime Minister then and the Labour Party had hardly started to shake itself into newness. But politics would never have been far from his thoughts. He is at heart a politician, as he has been since student days.
"He's got the politics bug," says an insider who watched him closely throughout his time in Hong Kong. "All the time he was here he was immensely interested in the minutiae of British politics." He could not have guessed until very recently how interested British politics would become in him.