Their four carers, who were operating the lock on the Leeds-Liverpool canal when the accident happened, dived in to try to save them. They have been treated for shock.
A 15-year-old boy and his father - who were on another barge alongside, also dived down twice - bringing three of the passengers to the surface.
Police, who described the boy's efforts as "heroic", said the three could not be revived. The fourth person, who had been trapped in the barge, was later recovered by fire crew wearing breathing apparatus.
The three men and a woman, who have not been named by police, were on a week's boating trip from a day centre in Cumbria. The accident happened at Stegneck Lock near Skipton, North Yorkshire, when the barge appears to have got caught on a lock gate and then collided with the other barge as it sank.
Pc Caroline Ainsworth, 25, who was giving counselling to the survivors, said: "To say they were distraught would be a massive understatement. They were beside themselves. They are simply too shocked to talk much about it, but one thing is for certain, they all tried their best to save the others.
"They have all told how they went into the water and tried to get hold of the people who were inside while the boat was becoming submerged under water. In the end it boiled down to them becoming caught and having to get out themselves."
The families of the deceased had been informed last night but police were not releasing details until after interviews with the four carers and the passengers on the other barge.
A spokesman for the ambulance service said: "We were told there were four disabled people on this boat and that it was sinking. When we got there there was still one person trapped inside. We tried to resuscitate three others on the river bank but sadly those attempts failed."
British Waterways, which manages the 127-mile canal, the country's longest, has launched its own investigation into the disaster. Its spokesman, Guy Lawson, said recent routine maintenance checks had revealed no problems at the lock.
He said: "It is the first fatality at this lock. The Leeds-Liverpool canal has always been very safe. It's such a shock."
The director of social services for Cumbria County Council, Mike Siegal, confirmed that the group was on a week-long holiday and had previous experience of canal boating.
In a statement he said: "The holiday included a total of eight people, four experienced members of staff who were supporting the four adults with learning disabilities from south Cumbria.
"The group has been on canal boat holidays in the past."Reuse content