The tycoon told his supporters at a meeting in London that he withdrew his candidacy with "much happiness and a little sadness."
"With happiness, because we've achieved what we set out to do in this constituency, and with a touch of sadness. There's a touch of Billy Liar in all of us ... I was looking forward to this campaign."
He paid tribute to Mr Clark, who was at the meeting, as "a proud and brave Englishman who shares our views and will fight our cause with grit".
"I will do anything I can to help him," added Mr Birley. "But I will, however, do most of my fighting in public - fighting that utterly gruesome David Mellor." Sir James is standing for Putney, against Mr Mellor.