He was eight when he caught his first sight of the imposing, 19th-century gothic buildings, isolated on the North York Moors, that housed the abbey, college and preparatory school. Even at that young age, Oliver, now in his forties, recalls being struck by the ascetic atmosphere pervading the institution.
Benedictine monks, many of whom had spent virtually their entire lives behind Ampleforth's limestone walls, ran, he says, an austere regime, where obedience and servitude meant everything and compassion was a rarity.
What Oliver could not have known then was the depths to which some monks, many educated at the school, were capable of sinking. He was entering a world of secrecy, where boys were beaten for sexual gratification and sexually abused. It would take three decades and an exhaustive police investigation before the prosecution of four monks who taught at the school and meted out some of that abuse. The fourth, Fr Piers Grant-Ferris, admitted last week to a catalogue of indecent assaults.
Oliver was one of his victims. "He had an obsession with little boys," says Oliver, whose name has been changed to protect his identity. "He was weird, slightly odd. The whole atmosphere of the school was."
Grant-Ferris carried out 20 indecent assaults during the 1960s and 1970s on 15 young boys, all under the age of 12. At first, Oliver thought his behaviour was merely odd. Child-like jokes would abound between the boys that Grant-Ferris was "pervy". Innocent of the seriousness of his actions, the boys told no one else. Grant-Ferris would regularly touch boys' genitals, he said. On one occasion he beat Oliver, while whispering, "I don't want to hurt you, but I have to", and apparently deriving sexual pleasure.
The first time Oliver admitted to himself that he had been indecently assaulted by Grant-Ferris was when he was visited last year by a police officer. In investigating allegations of assaults in the 1990s, police had discovered the abuse went back further than that.
"It was quite a release, liberating even. At the time, I wasn't in floods of tears or breaking down. Before then, I don't think I had given it any gravity. It is only now that I am getting to grips with my 'false' self.
"To have a monk take you to a room, strip the lower half and beat you was what was expected if you had been naughty.
"If a boy was unwell at night, and had to see someone, they would see him. He had a particular interest in taking a boy's temperature up the bottom. And then there was anything that went from there.
"I have best friends whom I grew up with from a young age, and I never knew what had happened to them. There are names on that list who are good friends ... I had no idea that they went through the same thing."
On leaving school at 18 he worked with the socially excluded and vulnerable. He has had a long-term girlfriend but the relationship is now over. He says his experiences have left him harbouring a deep-seated distrust of the public school system. He refused to attend a large reunion earlier this year, and avoids "old boy" events. He hopes his speaking out will encourage others to do the same.
Police believe that the four Ampleforth offenders were acting independently. But in an interview with The Yorkshire Post, Detective Superintendent Barry Honeysett, who led the investigation, said that later abuse cases could have been avoided if the school had informed the police about the allegations against Grant-Ferris when they first knew of them in 1975. Grant-Ferris has been bailed until January, but has been warned that he may face a long jail sentence.
Fr Cuthbert Madden, elected abbot in February, insists that "a great deal has changed" in the school since 1975, that it now has safeguards for child protection "almost unrecognisable" from those of the 1970s, and that it is regularly inspected by social services.
Oliver is sure that the late Cardinal Basil Hume, who was abbot at Ampleforth at the time of the offences, knew of the abuse. "The monks themselves knew full well what they were doing. They were damaged people themselves. I don't believe for a moment that Hume didn't know as well."
Oliver does not believe that prison is necessarily the right place for his abuser, but he wonders whether Ampleforth has fully addressed its past. He praises the appointment of a child protection officer, but says that the school would need to "wipe the board and start again" to ensure that the culture of abuse is eradicated once and for all.