"We're going to the left," came the reply. But after a moment's reflection, the chauffeur corrected himself: "No, we're going to the right."
The revised route was to Sir James's satisfaction. "Oh, good," he said, smiling at the cameras.
The Conservatives were, after all, "dead" and he was, indeed, heading for the right.
Speaking at a "special" - that is, his first and last - press conference at a London hotel, Sir James declared that it was not the Referendum Party that had killed off the Tories. Rather, the Conservatives had committed suicide.
"The policies they've adopted fly in the face of democracy," he said. "That was bound to kill them."
Earlier in the day, Sir James had learned of a death in his own party. David Crabtree, who was standing for the Referendum Party in Pudsey, in Leeds, announced he was backing the Tory candidate, Peter Bone, because of his Euroscepticism.
Clearly irritated by Mr Crabtree's last minute decision, Sir James said: "I think he made a mistake because I think that no matter who wins in Pudsey the Conservative Party will not win and that the key thing in terms of the future is a referendum."
He added mischeviously: "The people of Pudsey can still vote Referendum - and who knows, Mr Crabtree might end up a member of parliament. Look, 230,000 people have joined us from other parties. When one leaves us, it's fair balance."
Even if the Referendum Party does not secure a single seat on Thursday, the show will go on, said Sir James. "We will be a conduit for mobilisation as we have been in the past few months ... I do believe that we need to continue as a movement so that if, in secret as in the past, politicians seek to give up our sovereignty, we will be there."
But Sir James did not have it all his own way. He was accused of introducing an "entirely new tone into British politics."
"I have attended your rallies," shouted a lone voice, "And the loudest cheer was given to the man at Alexandra Palace who suggested that electrodes should be screwed to the sexual organs of Sir Edward Heath."
"I see you're extremely unfamiliar with British politics," replied Sir James, patronisingly. "We draw from all the parties ... ". Mr Roth butted in: "Has anybody suggested electrodes?". "Shut up a second," snapped Sir James. "I suggest you go back to Austria and deal with your own rightwing problems."
Lord McAlpine made his learned contribution to the debate: "It was a howl of pain, not a howl of delight."
In response to comments earlier in the day by Kenneth Clarke, Chancellor of the Exchequer, that the influence of millionaires in this election campaign had been over the top, Lord McAlpine's wit ran wild: "I assume he's referring to Michael Heseltine."
Sir James struck a more serious note. "One of the problems of the whole construction of Europe has been the massive propaganda campaign used by Euro federalists out of tax payers' money." His pounds 20m was peanuts in comparison with the pounds 220m - or more like pounds 500m - ploughed in from "the other side".
He said he didn't have "the slightest idea" how many people were going to vote for the Referendum Party, although he took the opportunity to share "a little bit of news" that reached him earlier in the day.
"I'm not suggesting that this is representative," he began apologetically. "Labour 36 per cent - leading," he began. "In number two position, the Referendum Party with 23 per cent. Conservatives 21 per cent. Lib Dems 11 per cent. Don't knows 9 per cent."
He was, in fact, quoting the results of the latest Labour Party canvassing returns in Gosport, Hampshire, not the national picture.
All the same, it was satisfying. "Now if the Labour Party is right," said a delighted Sir James, "I strongly recommend the Conservative Party candidate withdraw."Reuse content