Mr Clegg abandoned Westminster convention that party leaders avoid criticism of the office holder to demand the exit of "a dogged defender of the ways things are".
"I do not think the Speaker should be made a scapegoat...for the individual failings of many MPs, he told BBC1's Andrew Marr Show.
"But equally I do not think we can afford the luxury of a Speaker, who is supposed to embody Westminster, who has been dragging his feet on transparency and greater accountability in the way MPs receive their expenses."
His intervention comes on the eve of the tabling of a motion of "no confidence" in Mr Martin and reports he is set to announce he will step down next year.
Mr Clegg said: "Convention is that political leaders, party leaders, do not talk about the Speaker.
"My view is that it is exactly that culture of unwritten conventions, unspoken rules and nods and winks that has got us into that trouble in the first place.
"I have arrived at the conclusion that the Speaker must go.
"He has proved himself over some time now to be a dogged defender of the way things are, the status quo, when what we need, very urgently, is someone at the heart of Westminster who will lead a wholesale radical process of reform."
On a personal level Mr Martin was "kind and courteous", he said.
"But I do not think he is now the right man for the job in actually leading the renewal of Westminster that we need. We need a fresh start."
He spoke out as Labour former minister Kate Hoey became the latest MP to publicly state that she would sign the motion, which is being tabled by Tory Douglas Carswell tomorrow.
She was among MPs on the receiving end of a series of Commons rebukes from Mr Martin last week which intensified calls for him to step down.
"MPs should sign, because let's all be frank and honest, Michael Martin is not up to the role," she said.
John Stonborough, the Speaker's former media adviser, told The Sunday Times that Mr Martin had vetoed radical reform of the expenses system and reacted angrily when challenged over his own claims.
"He reacted extremely violently," he said when challenged that his decision to claim second-home allowance while living in a grace-and-favour property did not "look good".
Officials were too scared of him to challenge his decisions, he suggested.
Urging him to go quietly, he told the newspaper: "We should not have to watch the humiliation of him being voted out of office. There are a number of dignified exits he could do. The longer he leaves it, the less dignified they become."
Writing in the Mail on Sunday, Mr Carswell, who wants a new Speaker elected by secret ballot to ensure a fair contest, said: "The Commons Speaker is supposed to embody the House. Alas, Mr Martin has come to exemplify the Commons in precisely the wrong way.
The Speaker ought instinctively to grasp that MPs need to live under the same rules they make and impose on the rest of the country, he said.
"Instead, Mr Martin presided over a system that saw politicians awarded tax-free income in the form of off-balance-sheet expenses.
"Rather than see this as the ethics of Enron, he made every effort to ensure the details remained off the balance sheet."
Radical change was needed across the board at Westminster, not just on expenses, he said.
"This agenda can only begin with the election of a new Speaker."
He concluded: "Let us all be frank and honest, and admit Michael Martin is not up to the role. Let us be prepared to sign the motion tomorrow and, if necessary, brace ourselves for the consequences.
"Until we are prepared to acknowledge how terrible things have become in Westminster, we will not be able to change. Change must start with a new Speaker."
Mr Clegg said he represented a "growing body of opinion" among MPs in calling for Mr Martin to go. The Commons needed a Speaker who was a "reformist to his or her fingertips".
He added: "Westminster is now engulfed by a political crisis the likes of which we haven't seen for generations. We need to now do something different, radically different, and I just don't think defenders of the status quo are the right kind of people to do that."
He predicted a "snowball effect" with more MPs speaking out against Mr Martin: "I think it needs to happen sooner rather than later, I think the Speaker needs to do the decent thing, recognise things have changed, that he is not the right man for the job, and move on."
But Labour peer Lord Foulkes claimed some MPs were looking to make Mr Martin a "scapegoat".
And he said that he believed the Speaker had no intention of "bowing to pressure" to quit before the next general election.
Writing in the Sunday Times, Lord Foulkes said: "Over the past few days one or two motley MPs have been calling on the Speaker to resign.
"Those who have joined the call are the usual suspects, including bitter former ministers and even one of the losing candidates when Martin was elected - evidently still smarting."
He added: "I believe the Speaker has been encouraged by expressions of support from across the political spectrum and has no intentions of bowing to pressure this side of the next election.
"It seems that some MPs are looking for a scapegoat and mistakenly think his departure would take the pressure off them.
"Nothing could be further from the truth and it is not going to happen."
In his article Lord Foulkes pointed out that the Speaker does not traditionally give interviews, adding that as a result of this "whenever he is caught up in any controversy his side of the argument is unable to be expressed".
Lord Foulkes also said it was his understanding that Mr Martin had produced a three page chronicle of events covering the period from July 2007 to the present day, which showed that criticism of the Speaker was "unjustified" and that "reforms suggested by him have been sabotaged by those now seeking to smear him".
Comments