The 87-year-old former Cabinet minister spoke out after it emerged that the Prime Minister may announce in his speech to conference tomorrow that one million better- off families should pay the new tax.
The move to target higher rate tax payers could come in as early as the next Budget in Spring.
More than seven million families claim the benefit, but ministers have long believed that many middle-class parents simply do not need it. Taxing it could raise up to pounds 2bn a year without sacrificing the principle of universality, ministers believe.
However, Baroness Castle said: "Why pick on child benefit? The principle behind a good welfare state is that the benefits are universal."
Although many middle class parents claimed that they were willing to give up child benefit, those same people were the ones who eagerly grabbed the Tory tax cuts, she added.
Baroness Castle said that she wasn't optimistic that the new party conference arrangements would allow delegates to oppose the plans.
John Prescott, the deputy Prime Minister, was interviewed on BBC1's Breakfast With Frost yesterday and asked if the tax would be brought in. He said: "I don't know. That's Tony Blair's speech. This will have to wait until Tony makes his speech."
Rodney Bickerstaffe, general secretary of Unison, the public services union, told the programme: "If we're talking about clawing back money on child benefit, the principle of income tax is where the money should be clawed back. We are a very rich nation."
Ken Livingstone, the MP for Brent East, also suggested the rich should pay higher income tax.Reuse content