Emmerson Mnangagwa will take over as leader of the party, Chris Mutsvangwa, the chairman of the Zimbabwe War Veterans Association confirmed.
Mr Mutsvangwa also said the party was beginning the process to remove Mr Mugabe as President of Zimbabwe.
Members of the ruling party's Central Committee danced, cheered and sang as Mr Mugabe was recalled.
Later on Sunday, the 93-year-old will meet with the army commander who put him under house arrest in a second round of talks on his departure after nearly four decades in power.
The party has given the President until noon on Monday to step down or face impeachment, cyber security minister Patrick Chinamasa said.
Grace Mugabe, the country's unpopular first lady known as "Gucci Grace," has also been expelled from the party and recalled as head of the women's league, party sources told Reuters.
Without the military's intervention, Ms Mugabe was expected to have replaced Mr Mnangawa as vice president, putting her in a position to succeed her husband.
Speaking before a special Zanu-PF meeting to decide Mr Mugabe's fate, Mr Mutsvangwa said the 93-year-old was running out of time to negotiate his departure and should leave the country while he could.
"He's trying to bargain for a dignified exit," he said.
The war veterans' leader followed up with threat to call for street protests if Mr Mugabe refused to go, telling reporters: "We will bring back the crowds and they will do their business."
Mr Mnangagwa, a former state security chief known as "The Crocodile," is now in line to head an interim post-Mugabe unity government.
On Saturday, hundreds of thousands of people flooded the streets of the capital Harare to call for the resignation of Mr Mugabe after 37 years in power.
Under house arrest in his lavish "Blue Roof" compound, Mr Mugabe has refused to stand down, even as he has watched support from his party, security services and people evaporate in less than three days.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies