Amnesty International says they were killed at the Porta Farm settlement, home to 10,000 people, as police moved in with bulldozers to level the area.
"At least three have died – including a pregnant woman and a four-year-old child – during a chaotic mass eviction," the human rights watchdog said.
"There are unconfirmed reports of a second child dying. Circumstances are not yet clear," it added. This would take the total to four. Two children were crushed to death last month in a similar demolition.
The latest evictions come despite the presence of a UN special envoy in the country to assess the crisis. So far the official, Anna Tibaijuka of Tanzania, has made no public statement calling for a moratorium on the evictions.
Zimbabwe's President, Robert Mugabe, has been accused of launching a pogrom of the urban poor to punish them for voting against him in the disputed March election.
Operation Murambatsvina, which means "drive out the rubbish" in Mr Mugabe's native Shona language, has been going on for nearly a month, displacing close to half a million of Zimbabwe's poorest people.
Food shortages in rural areas, in conjunction with extremely high rates of HIV infection, are killing thousands of people every week. Life expectancy in what was until recently one of the richest nations in sub-Saharan Africa, has plummeted to 33.