Sir David Ramsbotham said people seeking entry into the country should be held in special immigration detention centres if it was decided they had to be detained while a decision was made.
At the moment, 938 immigrants are being held in detention and just over half are kept in prison establishments, rather than special centres.
In a report written after an inspection of Haslar Holding Centre, Sir David said he had "serious concerns" about keeping detainees in prison conditions. Although Haslar, in Hampshire, is a holding centre rather than a prison, it is run by the Prison Service under prison rules.
Sir David praised the centre's staff, saying they were working with humanity and care, but warned it was "unacceptable" that rules intended to apply to convicted prisoners were being extended to detainees. Prolonged periods of detention were "very stressful and can have an adverse effect on the mental health of a detainee", Sir David said.
He added: "I do not believe that asylum seekers or immigration detainees should normally be held under Prison Rules unless they have committed a crime which makes such detention appropriate."
In recent proposals to shake up the immigration system, the Government said it would rather hold detainees in immigration centres and planned to raise the number of spaces available in such centres.
But it predicted that prisons would still have to be used for reasons of "geography, security and control".
At the moment only about 1.5 per cent of applicants who might be liable for detention are held at any one time.
People are held if they are awaiting deportation, have entered the country illegally or if it is believed there is a danger they will "disappear" from the immigration system.
The Home Office minister Lord Williams of Mostyn said the detention of some immigrants was essential. "Detention is a necessary part of our immigration control system. However, we are committed to processing claims faster and pursuing alternatives."
The chief executive of the National Association for the Care and Resettlement of Offenders, Helen Edwards, said: "These places are wholly inappropriate for people in a distressed and vulnerable state."Reuse content