Maverick academic Philip Zimbardo says we are all capable of evil. Is he right?

First he devises an experiment that turns peace-loving students into brutal thugs. Then he defends a soldier accused of abuse at Abu Ghraib. Philip Zimbardo explains why his life is dedicated to darkness

On 28 April 2004, Philip Zimbardo was in Washington for a conference. The TV was on in his hotel room and photographs of the abuses carried out in Abu Ghraib prison in Iraq by US servicemen and women flashed across the screen. The images are ingrained in our psyche now, but then they were new. Naked men stacked in a pyramid with soldiers grinning alongside. A female soldier leading a prisoner around on a dog lead. Prisoners forced to simulate sexual acts on each other. A prisoner in a hood balancing precariously on a box in the belief he would be electrocuted if he moved. Like millions of others, Zimbardo was deeply shocked by what he saw, but for the professor emeritus of psychology at Stanford University, California, there was a disturbing element of familiarity.

"I had taken similar images myself 30 years earlier," he says. "And by similar, I mean prisoners with bags over their heads, prisoners stripped naked, prisoners made to do sexually degrading activities. It was very disturbing. [The scenes at Abu Ghraib] recreated emotionally the horrible things I not only saw but that I allowed to continue to happen." ' The images he is referring to came from one of the most infamous episodes in American academic history, the Stanford Prison Experiment – a study Zimbardo led in 1971 into the psychological and behavioural effects of imprisonment that swiftly descended into scenes of cruelty and degradation.

Zimbardo hoped he would never see Americans behave so abominably again. The shock of the Abu Ghraib scandal three years ago dashed that hope – and prompted the then-71-year-old to come to the defence of one of those accused of the terrible crimes committed in the Iraqi prison.

What took place on a peaceful Californian university campus nearly four decades ago still has the power to disturb. Eager to explore the way that "situation" can impact on behaviour, the young psychologist enrolled students to spend two weeks in a simulated jail environment, where they would randomly be assigned roles as either prisoners or guards.

Zimbardo's volunteers were bright, liberal young men of good character, brimming with opposition to the Vietnam war and authority in general. All expressed a preference to be prisoners, a role they could relate to better. Yet within days the strong, rebellious "prisoners" had become depressed and hopeless. Two broke down emotionally, crushed by the behaviour of the "guards", who had embraced their authoritarian roles in full, some becoming ever-more sadistic, others passively accepting the abuses taking place in front of them.

Transcripts of the experiment, published in Zimbardo's book The Lucifer Effect: Understanding How Good People Turn Evil, record in terrifying detail the way reality slipped away from the participants. On the first day – Sunday – it is all self-conscious play-acting between college buddies. On Monday the prisoners start a rebellion, and the guards clamp down, using solitary confinement, sleep deprivation and intimidation. One refers to "these dangerous prisoners". They have to be prevented from using physical force.

Control techniques become more creative and sadistic. The prisoners are forced to repeat their numbers over and over at roll call, and to sing them. They are woken repeatedly in the night. Their blankets are rolled in dirt and they are ordered painstakingly to pick them clean of burrs. They are harangued and pitted against one another, forced to humiliate each other, pulled in and out of solitary confinement.

On day four, a priest visits. Prisoner 819 is in tears, his hands shaking. Rather than question the experiment, the priest tells him, "You're going to have to get less emotional." Later, a guard leads the inmates in chanting "Prisoner 819 did a bad thing!" and blaming him for their poor conditions.

Zimbardo finds 819 covering his ears, "a quivering mess, hysterical", and says it is time to go home. But 819 refuses to leave until he has proved to his fellow prisoners that he isn't "bad". "Listen carefully to me, you're not 819," says Zimbardo. "You are Stewart and my name is Dr Zimbardo. I am a psychologist not a prison superintendent, and this is not a real prison."819 stops sobbing "and looks like a small child awakening from a nightmare", according to Zimbardo. But it doesn't seem to occur to him that things are going too far.

Guard Hellmann, leader of the night shift, plumbs new depths. He wakes up the prisoners to shout abuse in their faces. He forces them to play leapfrog dressed only in smocks, their genitals exposed. A new prisoner, 416, replaces 819, and brings fresh perspective. "I was terrified by each new shift of guards," he says. "I knew by the first evening that I had done something foolish to volunteer for this study."

The study is scheduled to run for two weeks. On the evening of Thursday, the fifth day, Zimbardo's girlfriend, Christina Maslach, also a psychologist, comes to meet him for dinner. She is confronted by a line of prisoners en route to the lavatory, bags over their heads, chained together by the ankles. "What you're doing to these boys is a terrible thing," she tells Zimbardo. "Don't you understand this is a crucible of human behaviour?" he asks. "We are seeing things no one has witnessed before in such a situation." She tells him this has made her question their relationship, and the person he is.

Downstairs, Guard Hellmann is yelling at the prisoners. "See that hole in the ground? Now do 25 push-ups, fucking that hole. You hear me?" Three prisoners are forced to be "female camels", bent over, their naked bottoms exposed. Others are told to "hump" them and they simulate sodomy. Zimbardo ends the experiment the following morning.

To read the transcripts or watch the footage is to follow a rapid and dramatic collapse of human decency, resilience and perspective. And so it should be, says Zimbardo. "Evil is a slippery slope," he says. "Each day is a platform for the abuses of the next day. Each day is only slightly worse than the previous day. Once you don't object to those first steps it is easy to say, 'Well, it's only a little worse then yesterday.' And you become morally acclimatised to this kind of evil."

The parallels to atrocities of this and the last century – atrocities we believe we are distanced from – are glaring. The behaviour of ordinary Germans under the Nazis. The slaughter of Tutsis by their neighbours, the Hutus, in Rwanda. How vulnerable are we to emulating such murderous behaviour in similarly extreme circumstances? Very, says Zimbardo. "We are unaware of how much our behaviour is influenced by situations, as the situations we are in are usually benign. The Stanford experiment looks at what happens when you put people in a totally new situation, where they don't have habitual coping mechanisms. So they look around. What are other people doing? What is the appropriate way to behave in this new place? If you are a guard, the appropriate way to behave is to demonstrate that the prisoners are powerless and you are powerful."

The seeds of Zimbardo's research were planted in his childhood. Born in 1933 in the Bronx, he lived there until he was 23. "It was and is one of the worst ghettoes in America," he says. "I knew good kids who went bad, who ran drugs and got in trouble, went to jail and got killed. And there were other kids who didn't. So I wondered, what makes good people go bad?"

The Stanford experiment caused a media storm and Zimbardo became a star, of sorts. He wrote about it and lectured on it, and life moved on. He married Christina Maslach and the couple had two daughters. His professional gaze turned to other themes, inspired by Stanford. He researched shyness – "a psychological prison" – and set up a clinic to treat it. He worked on time perspective ("a day in the experiment began to feel like several days"). And in 2002 he was elected president of the American Psychological Association. Stanford had long been "laid to rest". But when he saw the Abu Ghraib pictures, the past was stirred up again.

"When the American military and Bush administration immediately distanced themselves by saying Abu Ghraib was the work of a few bad apples, I was suspicious," he says. "I knew that in the Stanford experiment, I began with good apples and that it was the place that corrupted them, so my hypothesis was that maybe these soldiers were good apples and it was the barrel at Abu Ghraib that corrupted them."

His response was twofold. First, he went back to the 12 hours of videotape he had from the Stanford Prison Experiment, reviewed it with his students, and made full transcripts. "I decided I really had to present it in great detail because the evil was in the words. It was in how the guards created a psychological system that crushed the prisoners."

Second, he agreed to appear as an expert witness for one of the defendants in the Abu Ghraib trial, Staff Sergeant ' Ivan "Chip" Frederick II. Frederick was the military policeman in charge of the night shift on tiers 1A and 1B, the site of the abuses, and features in some of the pictures.

Zimbardo threw himself into the case, counselling Frederick and his wife. He sought out and examined Frederick's records (unstinting dedication in the service of his country and family). He had Frederick undergo psychological tests (good man vulnerable to isolation, strong desire for approval). He investigated Abu Ghraib, and the conditions there. He made presentations to Frederick's military trial – but to little or no effect: Frederick was convicted of five charges of abusing Iraqi detainees and received an eight-year sentence.

"There was a real injustice," says Zimbardo. "Colonel Larry James, a psychologist sent to Abu Ghraib to fix it, said that 50 times he was within 100 yards of being blown up or shot. It was 130 degrees. There was faeces everywhere; rats running around; 1,000 prisoners, many naked; people screaming. It was hell." Frederick, he explains, worked 12-hour shifts in these conditions without a night off in 40 days. When he finished his shift he would sleep in a cell, "so he was always in prison". Not once in three months did a senior officer come down to his area, says Zimbardo.

In Level 1A, the interrogation centre run by the CIA, military intelligence and civilian contractors were looking for information on the insurgency and getting nowhere. They put pressure on Frederick's team to "take the gloves off and soften the prisoners up", says Zimbardo. There is some evidence, he adds, "that the early pictures were staged" so they could show them to other detainees before their interrogation. "Once they got permission to break those prisoners and take those pictures, you have unleashed the dogs."

The sexual abuse was the next stage. "When you have a unit of men and women soldiers, and hundreds of prisoners running around naked, there is a sexual dynamic. Some of those soldiers are having sex with each other, and some of the people arrested were prostitutes. It was a lawless hell.

"When you ask Chip why he did these terrible things, he says 'I don't know what came over me.' He had lost his reason, perspective and judgment. If you had any sense of reason, you would never put yourself in the picture. You are making yourself accused. What were they thinking to do that? The answer is that they weren't thinking."

With Abu Ghraib, Zimbardo extended his theories beyond the "situation" to the "system" that created it. "I would point the finger as high as [President] Bush. With his excessive focus on fear of terrorism, with the lawyers who legalise the definition of torture, he creates the system."

In both situations – and in others where abuse escalates – Zimbardo isolates common factors. The first is "deindividuation": the perpetrators become anonymous and stop acting as individuals. The guards in Abu Ghraib were in the habit of removing identifying details on their uniforms; the Stanford experiment guards wore mirrored sunglasses that hid their eyes. The second factor is dehumanisation: the prisoners in both situations were seen as hostile and "other". Their physical condition was poor, they smelt, and they were often naked – like animals. Third, such abuse requires bystander apathy – the failure of the majority who may not be actively involved to do anything to stop it.

The pressure to go along with the escalation of abuse is huge, says Zimbardo, and would claim most of us. "We all have this egocentric bias to say, 'I would be the hero, I would blow the whistle,'" he says. "But other things being equal, you would do what they did. Though there are always a few who resist. And that is the hope of humankind."

Usually the whistleblower is an outsider, who views the situation with fresh eyes. In his experiment it was Christina. At Abu Ghraib it was 24-year-old reservist Joe Darby, who was shown images of the abuses by a fellow soldier. At first he thought they were "pretty funny", but found he "could not stop thinking about it". He said that what was happening "violated everything I believed personally". After three agonising days of feeling torn between loyalty to his friends and to his moral conscience, Darby blew the whistle.

Zimbardo is now researching heroes such as Darby, "ordinary people who do extraordinary things when other people are doing bad or doing nothing". His findings so far indicate that there is nothing in background, belief or personality that would predict who these people will be. The only certain thing, he says, is that "heroes are always deviants": they always question authority. "We just did a study in Italy, where we put people in a situation when authority makes you do something bad, to see who defied. Nothing we measured before would have predicted the outcome. All the people who defied could say is that they were more concerned about this other person than about the experiment or the authority. They showed an ability to empathise."

Most of us live in happy denial; we are never tested. I wonder how it must feel to have been tested as Zimbardo was, and to have been found wanting. He got caught up in the Stanford experiment; enmeshed in the values of the false system he had created, manipulative in protecting it, seemingly impervious to the suffering in front of him.

As a teenager Zimbardo read JM Barrie's Admiral Crichton – the tale of the silent, honorable butler transformed into a leader when the family he works for are marooned on a desert island, had a big effect. "It was one of the early awarenesses I had of the power of the situation." But this awareness did little to affect his own behaviour. "Did [Stanford] change my sense of myself? Absolutely. I grew up with the police as the enemy, they're never for you. And I become that thing that all my life I am against.

"At the time, you are not shocked, you are embedded in the situation. There's no guilt, no remorse, because there's no perspective. Afterwards, of course, I was ashamed. I had changed within five days. That is the more powerful lesson of the experiment than how the guards got into it: they were kids, I was a grown-up."

The Stanford Prison Experiment is one of very few academic studies to have made it into the public consciousness. I ask Dr Peter Banister, head of psychology at Manchester Metropolitan University and an expert in prisons, how it is regarded within the discipline. He says it is seen as important, but not necessarily for its findings. "In hindsight it is viewed as being ethically dubious; it is regularly used now in exercises concerning ethical problems in psychology."

"At the time there was no criticism," says Zimbardo. "It was a different era. If I had done the study right now, there is no question that I would be sued by every guard and every prisoner. These studies are in ethical time capsules. They cannot be done in a legitimate way now. In fact, the pendulum has gone so far in the other direction, you can't even ask questions that might be stressful. So my feelings are mixed. Do I want to be part of an infamous study? No."

Zimbardo and his wife live in a four-storey house overlooking San Francisco Bay, on the famous Lombard Street. Tourists flock outside, and it is all brightness and light, a long way from evil. The lighthouse beam from Alcatraz Prison, out in the bay, shines into his living-room. "[Stanford] was a little week-long study," he says, "but it has affected my whole life."

'The Lucifer Effect' by Philip Zimbardo (Rider, £8.99) is published in paperback on Thursday

Our dark materials: When the Lucifer Effect strikes

The Milgram Experiment

In the 1960s, Yale psychologist Stanley Milgram explored people's willingness to inflict pain on others when ordered to do so. A subject was ordered by a "teacher" in a lab coat to administer increasingly powerful electric shocks to a "learner" whenever the learner answered a question incorrectly (teacher and learner were role-playing and no shocks were administered). When the learner started screaming from pain, many subjects questioned whether to carry on, but most continued after being assured they would not be held accountable. Milgram carried out his first experiment three months after the start of the trial of Nazi war criminal Adolf Eichmann. As Milgram put it, "Could it be that Eichmann and his million accomplices in the Holocaust were just following orders? Could we call them all accomplices?"

Deepcut barracks deaths

Between 1995 and 2002, four young army recruits – Sean Benton, Cheryl James, Geoff Gray and James Collinson – were found dead as a result of bullet wounds at Princess Royal Barracks in Deepcut, Surrey. The army said the wounds that caused their deaths were self-inflicted while on sentry duty. A BBC Panorama documentary in 2002 then put forward what it claimed was evidence of systematic beatings and abuse at the barracks, alleging that the camp was "dominated by fear, violence and sexual harassment". The possibility of a public inquiry was ruled out. An independent review in 2006 criticised the mistreatment of trainees at Deepcut, but concluded that the deaths were self-inflicted.

Russian Army brutalities

In 2006, Sergeant Alexander Sivyakov of the Russian Army was convicted of beating an 18-year-old private in his care so badly that his legs and genitals had to be removed. The case outraged Russian society, leading to calls for the government to end its policy of national service. In 2004, the Human Rights Watch published a lengthy report about abuses in the army, in which it is alleged that senior NCOs brutally "initiate" new recruits by dedovshchina ("The rule of the grandfathers"). According to The New York Times, at least 292 Russian soldiers were killed by dedovshchina in 2005 and there were 3,500 reports of abuse. The Russian military concedes that 16 of the recruits were murdered, but says the remainder committed suicide. The office of Russia's chief military prosecutor claimed the army was working hard to stamp out abuse.

Welsh children's homes abuse scandal

In February 2000, the North Wales children's homes inquiry, headed by Ronald Waterhouse QC, led to the publication of "The Waterhouse Report", which uncovered one of the 20th century's biggest British child-abuse scandals. It alleged that at least 650 children had been abused in homes in Clwyd and Gwynedd in the 1970s and 1980s. The report also suggested the Welsh Social Services Inspectorate had carried out checks on only five homes over seven years, and noted that complaints were generally dismissed, police investigations poorly carried out and appeals to government ministers ignored. In the wake of the report, it was estimated that 100 people were prosecuted for abusing children in their care, and around 50 more were investigated. At least 12 of the abused children committed suicide, while many more lived troubled lives. Victims included Steven Messham, who claimed to have been abused by over 40 people at the Bryn Estyn children's home in Wrexham. He has since spent much of his life in psychiatric hospitals. Luiza Sauma