Jimmy Savile regularly took teenage girls to a hospital block alone, a former porter has claimed.
Terry Pratt said the late Jim'll Fix It star was frequently handed a key to the nurses' accommodation building at Leeds General Infirmary during the late 1980s.
The claim comes as Scarborough council announced it is to consider posthumously stripping Savile of his freedom of the borough, which he was awarded in 2005.
Councillors are to put forward a motion to wipe his name from Scarborough's roll of honour following claims that he abused as many as 300 victims.
Mr Pratt told the BBC that Savile would arrive at Leeds General Infirmary with the girls in the early hours of the morning and then leave before dawn.
He said: "He would go up and the lad on the desk would say 'Here's the key, Jim, make sure I get it back'.
"He'd take the key and would walk out and the two women would follow him towards the nurses' home.
"He was going into a property he had no right to go into. He wasn't a doctor and he wasn't a nurse."
Mr Pratt said he became suspicious when Savile began arriving in the middle of the night with different girls who seemed "star-struck" and were "not streetwise".
"You'd see their eyes watching him - everything he did," he said.
But nobody questioned the late presenter's behaviour at the time, he said, adding: "We looked at him as a sort of film star, a cult figure."
He said the celebrity, who was a volunteer porter and fundraiser for the hospital, would make several late-night visits a month where he would ask for the key to the accommodation block, spend a few hours there and then leave at 5am.
Leeds Teaching Hospitals NHS Trust said it was "shocked" by the claims surrounding Leeds General Infirmary and has vowed to help the Metropolitan Police with inquiries into the alleged abuse.
A spokesman for the trust said: "We continue to be shocked by each new allegation. It is important that they are investigated properly.
"Once again, we urge anybody who has any concerns to contact us so that we can pass information on to the police or to contact the police directly via the NSPCC helpline, 0808 800 5000.
"The trust is in contact with senior detectives from the Metropolitan Police and we have indicated our intention to help with their inquiries.
"If there are any issues which need to be addressed following the police investigation, then we will take action."
Meanwhile, councillors in Scarborough, North Yorkshire, have called for his name to be immediately removed from the honour board for freemen of the borough until the Metropolitan Police concludes investigations.
Scarborough Borough Council will then decide whether to permanently wipe his honorary link to the seaside town, where he owned a second home and was buried overlooking the sea.
Councillor Tom Fox proposed the motion, saying: "This council wishes to send its heartfelt support to, and acknowledge the courage of, those who have come forward having suffered horrendous abuse at the hands of the alleged serial predatory sexual offender Sir Jimmy Savile.
"In addition, this council, in further acknowledgment of the serious predatory sexual allegations spanning over six decades, agrees that if the council had been aware of such revelations at the time of Sir Jimmy Savile's nomination for honorary freeman of the borough of Scarborough, the council would have refused it.
"It is therefore proposed that his name be removed forthwith from the honour board for freemen of the borough pending the final report from the Metropolitan Police, when this council will make a permanent decision in relation to the matters referred to in this motion."
Mr Fox said the council supported the removal of street signs and a memorial plaque that celebrated the late presenter's links to the town, and his headstone, which was recently removed and broken up at the family's request.
Historically, anyone made a freeman was exempt from tolls and given special privileges. It is still awarded to "persons of distinction and any persons who have rendered eminent services to the borough".
Scarborough has granted freeman status to 20 individuals and groups since the local government reshuffle of 1974, including playwright Sir Alan Ayckbourn, retired boxer Paul Ingle and the Yorkshire Regiment.
The motion to remove Savile's name from the honour board will be considered at a full council meeting next Monday.
Scotland Yard has launched an investigation into the television and radio star's activities, and he is now believed to have been one of the UK's most prolific abusers, with about 300 possible victims.
Detectives are following 400 lines of inquiry as part of the investigation while the BBC has launched an investigation into the culture and practices at the corporation in the era of Savile's alleged sexual abuse.
It is also looking at the decision-making process that saw a Newsnight investigation into Savile's activities shelved.
Deputy Prime Minister Nick Clegg has said "heads will need to roll" at the BBC if it is discovered that abuse was ignored.
"Serious questions need to be asked and if after we find out what's happened, it's clear that people have turned a blind eye or, worse still, connived with it, then of course they're going to have to be held to account and - if that turns out to be the case - heads will need to roll, of course," he told ITV's The Agenda.
Earlier this week it emerged that Savile was barred from any involvement with the BBC's Children In Need charity.
Sir Roger Jones, a former chairman of the charity, said he had been uncomfortable about allowing Savile to have any association with their work.
Although he had "no evidence" that Savile was up to anything, he said he behaved strangely, adding: "I think we all recognised he was a pretty creepy sort of character."
Britain's most senior police officer said if allegations about Savile had been linked while he was alive, they would have exposed "a pattern of behaviour".
Metropolitan Police Commissioner Bernard Hogan-Howe said police and other organisations had not connected a number of separate claims made about Savile's allegedly predatory actions.