Then, when I enter his actual office, he looms towards me out of such a swirling, smoky smog he's like someone getting off a train in a David Lean movie, only he's not very Omar Sharif, because he's quite fat and round and not much of a sex god, frankly.
Ken, I say, I'm something of a smoker myself. (I've even worked out a way of doing it in the shower, patent pending.) But this is terrible.
I ask his secretary how she bears it. "You get used to it," she says. "I warned Debbie at the start that I smoked in my office," says Ken. "Do the windows open?" asks the photographer. "Everyone at the Treasury got used to a smoky room," says Ken who, I think, is beginning to feel quite ganged-up on by now. He then says he would open a window, but is new to this office and hasn't quite worked out how.
Maastricht Treaty documents. Window engineering. Not subjects that have ever interested him greatly.
He is now just a backbench MP and, as such, had to hand back his grander offices. But he truly didn't mind, he says, and may even like this one, which is over the road from the Commons, rather better.
"The rooms I had in the House were near the Speaker's chair, and people were always dropping in, and you could never get anything done. Here, you can retreat only too effectively and can even forget to go across to the Palace at all, which is no great shame."
I don't think Ken is fed up with politics. Not a bit of it. But I do think he might be fed up with recent politics. Or, as he puts it at one point: "I think if I'd had to attend one more meeting about the precise words we were going to use to describe our attitude to the single currency I'd have gone mad. MAD!"
All in all, he's been at the very heart of British government for a good 16 years, but now isn't. It must feel strange, I say. He says it feels very strange indeed but, surprisingly enough, he is finding it pleasurably strange. "Last night, I put down my book [George Shultz's memoirs] because I discovered A Fish Called Wanda was on the television. Not long ago, I wouldn't have been doing any of those things. I'd have been doing red boxes." Did he enjoy A Fish Called Wanda? "Oh yes. I thought it very good."
So is this what he'll be doing from now on, watching more telly? "Sorry, Chief Whip, I can't possibly come in to vote. It's my Emmerdale night." No, probably not. "I don't propose to semi-retire. I have a very low boredom threshold, so I get bored if I don't work properly." He's had a lot of offers from banks and suchlike, he says, which he is going to consider over the summer. He'll make a decision when he gets back. But you'll be very busy going on these `bonding' weekends for Conservative MPs that Mr Hague has announced when you get back, won't you? So that you can relate to your colleagues better? "Hmmph," goes Ken.
You're not looking forward to it, then? "There's nothing wrong with the idea. I just won't be playing prank-ball myself," he says.
Poor Ken. Nothing's ultimately worked out the way he would have liked.
He's wanted to be Prime Minister since he was seven. He stood up in class at primary school and said so. He devoted his entire adult life to this end, then never even got to be leader of the party. He's not even in the Shadow Cabinet, although he could have been. "William offered me deputy leader. Had I been interested, I'd have gone on to say: deputy-leader to do what? I wouldn't have wanted to be a John Prescott-type deputy, sent round the beaches of England."
He must, I say, be crushingly disappointed at the way things have panned out. No, he insists, he isn't. "I'm not broken hearted. It's not in my temperament. Yes, I would have liked to have been Prime Minister. And, yes, I would still like to be Prime Minister." Still? "I am only 57," he says. "And I have more political experience than anyone else in the party. Who knows what's going to happen in the next few years?"
Only to be expected I suppose.You don't get to to the level he's been at for so long without being very ambitious. And people who are very ambitious don't, as a rule, suddenly stop being so. Does Ken really possess the jolly equanimity he would like us to think he does? Or is it all just front? Is he really a good bloke? Or does he just put on a good show? I think he's a bit of both, myself. But you can't dislike him for it, not least because the show is always such an engaging one. The thing about you, I later tell him, is that even though you're a Tory most people think you'd be OK to bump into down the pub.
"That's because most of them have. Ha! Ha!" he says.
Kenneth Clarke was a political animal from very early on. He was brought up in Nottingham, the son of a man who was upwardly mobile before the phrase had even been invented. A colliery electrician, Kenneth Clarke senior went into repairing watches and ultimately ended up owning three jewellery shops.
His father, he says, was a very easy-going, sociable, popular man, whereas his mother, Doris, was a different kettle of fish altogether. In a recent biography Ken's younger brother, Michael, was quoted as saying that Doris was an alcoholic, an unhappy, temperamental woman who spent a lot of time in her bedroom drinking gin.
Ken, however, refutes this hotly. "It's just not true. My brother fantasised it all. Why? I don't know. I am not very close to him, as you have probably gathered. My mother was a much less happy character than my father. She was more tense, more nervous, more argumentative and had fewer friends. And perhaps, after we left home, she did drink more than was good for her ... but not an alcoholic, no." Does he think she was a depressive then? "She was just quite complicated." Did she ever receive psychiatric help? "Not that I can recall, no."
Whatever, Kenneth junior was exceptionally clever. He could read well before school age. At school, he was always top in pretty much everything.
However, his parents being bright but not well-educated, there were never any books about the place. So, instead, he took to reading their Daily Mail from front to back every day. At an age when you and I were still pushing peas up our noses, he could tell you who was in the Cabinet and what bills were due to be heard.
At 11, he won a scholarship to a public school, then went on to Cambridge to study law. Here, he firmed up his political beliefs - became chairman of the Cambridge Conservative Association and all that - and met Gillian, his wife-to-be. He thinks they met at a barbecue held on Midsummer Common. He thinks they might have gone "to something French and grainy at the cinema" on their first date. He can't remember exactly. Ken's never much cared for detail. He's always been a doer, not a thinker. Whenever his driver used to ask him which boxes he wanted to take home that night, he would say: "Oh, any two." He never read the Maastricht Treaty because he couldn't be bothered. He was always a quick master of briefs, but never a diligent one.
Gillian Clarke is a medieval historian who is very clever, by all accounts. Anyway, I tell him she has always struck me as a fabulous sort.
Unlike other minister's wives, she's never gone in for the John Frieda hair-dos and little Windsmoor or Jaeger suits, has she? No, she most certainly hasn't, he cries. "She doesn't dye her hair. She's not interested in clothes. If she came down one morning in haute couture, I would be seriously worried about her." Neither, he continues, have ever succumbed to any of that Colour Me Beautiful nonsense. "During the election I was accused of having been to see an image consultant because my hair looked different. Well, the only thing different about it was that I'd washed it."
Anyway, he was first elected an MP in 1970, when he must have thought: "Right, I'm on my way to being PM now." Although, of course, he wasn't. He wasn't even on his way to leading the party, although it hasn't been much of a party to lead as of late, has it? Unfortunately, he can't argue with that, he says. It was, he reckons, Margaret's departure followed by the Danish referendum and then Black Wednesday which did them in. "The Euroscepticism, which had previously been quite subdued, suddenly leapt into life. The Eurosceptic revolt destroyed the Government." He could, he says, well understand Labour's victory.
Trouble is, we're all going to come to regret it, or so he insists: "I like Tony Blair. I rate him. I think he's very able. But I tell my European friends, `don't expect any heroics from Tony Blair.' I don't know what he's going to be like when the going gets tough. Gordon has already taken some tough decisions, yes. But, unfortunately, they were the wrong decisions."
How did he rate John Major? I ask. "He had all the qualities of charm and likability and being human, but no luck. He really was the unluckiest PM this century. How could we have predicted cows suddenly getting this bizarre disease?" Or the Hamiltons turning out to be not only a disgrace, but a whinnying disgrace. "I think they were, perhaps, a little noisier than was wise." Or Jonathan Aitken doing what he did? "I know him well. I like him. But I don't for the life of me understand what he thought he was doing."
With regard to the leadership contest, I say what I think many people believe, that he probably talked a good game right up until his Faustian pact with John Redwood, a pact that looked the opposite of wise.
Wise didn't come into it, he replies. "Firstly, it was essential. I was only two votes ahead at the end of the second ballot. The decision was in the hands of the 36 Redwood supporters."
But, even if you'd pulled it off, you and John couldn't have seriously made it work, could you? "Yes! Yes! We could have. We didn't need to do anything other than agree that the single currency was an open question. We both believed we could reunite the party. But then his supporters took off.
"Apart from Teresa Gorman. She voted for me and wrote me a nice note afterwards." Have he and Gillian had her round to supper yet, as a way of saying thanks? "Ah. No."
Ken and Gillian are off to California for the summer. She likes botany. He likes bird-watching. They'll travel about a bit, stopping in places which give "good botany and good birds".
Ah, I say, so you like the wife to scrabble about in shrubs at your feet while you gaze importantly into the sky? "Precisely," he whoops, well- pleased. Is Ken just a showman? Perhaps. But it's a terrific show.