The proposal by the Gracewell Institute in Birmingham was dropped this month after Baroness Faithfull, a former trustee, refused permission for her name to be given to the planned unit.
She wrote to the chief executive saying she thought it would be 'unwise' to have sexually abused children so near to sex offenders.
But the institute, where four priests were sent for treatment after accusations that they sexually abused teenagers, defended the original plan. Ray Wyre, the director, said putting the children's centre next to the sex offenders' unit was 'progressive'.
He said that in some hospitals abused children and sex offenders had to share waiting rooms. 'Our plan was to make sure nothing like that happened.'
But the plan, which included a first-floor extension joining the sex offenders' unit and the children's therapy centre, was condemned by Concerned Residents, a local community group.
Its chairman, Sean Wheatley said it was a 'bizarre' plan which would have put vulnerable children at further risk.
Baroness Faithfull, who is chairwoman of the all-party parliamentary group for children and involved with several children's charities, said: 'I let it be known I didn't think it was a good idea to have children on the site.
'I formally wrote on 27 June saying we considered that the siting of a child and family centre so near is perhaps unwise.'
She said: 'Any suggestion of there being a hostel or treatment centre for children is completely dropped and out. In any case I have withdrawn my name from it.'
Gracewell includes a hostel and treatment clinic for 21 sex offenders. It was set up by Mr Wyre, with almost pounds 1m from a Birmingham businessman, Trevor Price, to carry out a pioneering treatment.
The Concerned Residents group has written to MPs and the Home Secretary expressing fears that the institute is not accountable to statutory authorities.
The concern prompted Birmingham city council's social services department to review Gracewell's activities. A department spokesman said the review recommended changes, but declined to give details.
Birmingham planners also investigated complaints about Gracewell and a spokesman said it was operating without proper planning consent. When Gracewell was set up it was given planning permission to carry out whole family therapy, which did not include treating sex offenders, the spokesman said. Earlier this year the institute submitted a planning application which is still being considered.
Yesterday, despite Mr Wyre's assurance that plans to set up the children's centre had been abandoned, planners confirmed an application was still being considered for it.
Mr Wyre said the original idea had been to use the additional centre to treat the children and partners of offenders at Gracewell. But he said: 'When it became clear that we could do much more with this centre and treat children not linked to Gracewell we decided not to go ahead with the plan to have it next door.'
Company records show Gracewell was originally set up as a commercial proposition. An initial auditor's report said the company had 'immensely profitable opportunities'.
This has caused concern among local residents, but Mr Wyre said Gracewell was now registered as a charity. He pointed out that Trevor Price, who set up the original company, had now resigned as chief executive.
Mr Wyre said: 'I think if private funding had continued and we had got very successful all of us would have had problems.'