The former Hong Kong governor is currently heading the independent commission into policing in Northern Ireland.
But he told BBC1's Breakfast with Frost programme that once his contract was up - in nine months' time - he might make a comeback into frontline politics.
Asked what he would do if he was offered a safe Tory seat, Mr Patten said: "I would say: 'It's extremely kind of you to have called. Please call again next July, or a bit later.'"
He admitted that he missed "political debate and argument" and his friends at Westminster, but ruled out a return immediately to the Commons, saying he had not decided what he would do after his Ulster job.
"I haven't even talked about it at home," he said. "I know I want to do something else in public service because those are the most interesting jobs around. But I don't know how that will play out."
In a recent interview he was also reported to have said he would not be averse to standing as a constituency MP despite his rather more elevated past and ambitions for the future. Mr Patten, a strong contender to become the Mayor of London, has also been tipped by some to become the leader of the Conservative Party.
Nevertheless, he had warm words for the current leader William Hague.
He said: "He has got to climb the mountain, but he has done extremely well. He does very well in the House of Commons every week, and sooner or later that is going to have an effect on his standing in the country."
But he advised him to take a leaf out of the late Labour prime minister Harold Wilson's book when the referendum on a European Single Currency comes, which could see Mr Hague and Mr Patten on opposite sides.
Mr Patten said: "I think what the Conservative Party has to do is to take a lesson from Harold Wilson, who in his time constructed a position for the Labour Party which enabled people to argue on different sides in a referendum campaign, while being on the same side in an election campaign."
His comments came as it emerged that Mr Hague may be considering a referendum of Conservative Party members on the single currency if next month's party conference looks in danger of being disrupted by pro-Europe MPs.