John Bostock, who resigned as a Conservative Party agent last summer to work for the new organisation, has now moved on again, to the UK Independence Party.
He said that in some parliamentary seats the Referendum Party did not have a single supporter. Sir James and his supporters plan to field a candidate at the general election in every seat where the sitting MP fails to back a referendum on Europe.
Mr Bostock, who was paid pounds 24,000 per year by the party, was a regional campaign manager, responsible for overseeing the fight in 71 seats. However, he became convinced that the party's appeal was severely limited and that its ambitious plans were based on "a complete and utter fallacy".
"The whole thing is a nonsense. Fifteen-thousand out of 58 million people is a drop in the ocean," he said of the number of pledges of support. "It is a nothing party ... The average Conservative Party association has about 1,000 members."
Mr Bostock's comments will lend weight to the arguments of those who say Sir James's project is implausible and that it will not catch the popular mood in Britain.
The former Conservative will now stand as a candidate in Preston for the UK Independence Party, which believes that Britain should withdraw from the European Union and that trade links with Europe should be renegotiated.
Yesterday he said the Referendum Party simply lacked organisation, and that there was no real co-ordination between its departments. A promised list of supporters had failed to materialise.
"It is a campaign run with amateurs. People are leaving all the time, for many of the same reasons - lack of organisation and disillusionment."
Mr Bostock accused Sir James of "back-tracking" on his original suggested referendum question - whether Britain should be governed from Westminster or Brussels. The Referendum Party leader now prefers a "multi-optional" plebiscite accommodating a "diversity of views".
A spokeswoman for the Referendum Party, Priti Patel, said it did not have members because it was non-political, campaigning simply on a single issue. Mr Bostock was "clearly slightly bitter" because he had only managed to recruit five candidates in the North-west. She said there were now 48 candidates ready to stand in the area. "We have replaced him with a very strong regional team."Reuse content