Jean Marie Le Pen, leader of the French National Front, and Alessandra Mussolini, a far-right Italian MP who is granddaughter of the wartime fascist leader Mussolini are due to address a fringe meeting organised by the British far-right organisation Western Goals.
Sir Norman Fowler, party chairman, said the party was unable to prevent the meeting taking place during the conference in Brighton next month.
Sir Norman said no such meeting would be advertised in the official party guide nor would it take place within the secure area surrounding the conference centre.
A Central Office spokesman said: 'Western Goals is not affiliated or associated with the Conservative Party in any way. The party has no ties either with the French party to which Monsieur Le Pen belongs nor the Italian party to which Miss Mussolini belongs.
'The Conservative Party has no request fronm Western Goals to hold a fringe meeting at the conference and has no power to prevent outside orgaisations from holding meetings in Brighton but we do disassociate ourselves from any such meeting which Western Goals might hold.'
After the Independent reported opposition to the proposed visit by Ms Mussolini the Anti-Nazi League was informed M Le Pen had also been invited and agreed to speak at the same meeting.
The group said: 'The ANL is shocked that the man who claimed that the Holocaust was 'a mere detail of history' is being given a platform at the Tory Party conference. The fact that the invitation to Le Pen comes from individuals associated with Britain's governing party will give aid and succour to Nazis in this country.
'His invitation should also be seen in the context of the worrying rise of the neo-Nazis across Europe.'