Sir Julian, who is stepping down at the next election, said his decision was based on objections to the MP for Ludlow (the constituency where he will spend his retirement), the Euro-sceptic, Christopher Gill.
However, Sir Julian said he would not back Labour or the Liberal Democrats.
He announced his decision in a newspaper article in which he fired a devastating broadside at the "less well-educated, deeply prejudiced and chauvinist" Tory party left by Margaret (now Baroness) Thatcher, and wrote off the chances of a Tory election victory.
The timing of the attack - on the day the Government unveiled its legislative package for the new session of Parliament - will infuriate John Major and senior ministers.
The left-leaning MP for Aldershot, who has been no stranger to controversy during a colourful political career, said he could not bring himself to vote for Mr Gill, who was one of nine Tories removed from the whip at the turn of the year.
Sir Julian said Mr Gill would take Britain out of Europe.
"Should any Tory who has been pro-Europe since the days of Harold Macmillan and a member of the One Nation group of Tories be obliged to vote for a candidate who not only lost the party whip in 1994, but ... does not seem to care whether his party wins or not?" Sir Julian wrote in the London Evening Standard.
"I shall sit on my hands. Blair's new Labour Party holds no attraction for me and nothing on earth would persuade me to vote for the Liberals."Reuse content