Cavendish suffered a broken shoulder blade when he was forced into the barriers by Sagan 120 metres from the finish of Tuesday's 207.5km stage from Monsdorf-les-Bains to Vittel, won by French sprinter Arnaud Demare of FDJ.
But Sagan and his team insisted he had not seen the Manxman as Cavendish tried to race up his inside, saying he was not at fault for the crash.
In a statement, the team said they had "applied for a redress" of Sagan's result, as the Slovakian crossed the line second behind Demare before the race jury intervened.
Sagan entered the Tour as the red-hot favourite in the points classification as he sought to win the green jersey for a record-equalling sixth consecutive year.
However Sagan's protest ends, Cavendish's Tour is over, along with the 32-year-old's hopes of adding to his 30 career Tour stage wins for another year.
Cavendish, who is four Tour stage wins shy of Eddy Merckx's all-time record, said: "I'm obviously massively disappointed to get this news about the fracture.
"The team was incredible today. They executed to perfection what we wanted to do this morning. I feel I was in a good position to win and to lose that and even having to leave the Tour, a race I have built my whole career around, is really sad."
Cavendish had spent three months prior to the Tour fighting back from a bout of the Epstein-Barr virus, only securing his Tour place at the last moment, and the early exit after promising signs will come as a bitter blow.
"My heart has gone to my feet," his Team Dimension Data sports director Roger Hammond said. "These riders are like my children. That guy has worked so hard to be here."
The incident happened moments after another crash had sent Team Sky's Geraint Thomas to the deck in the race leaders' yellow jersey. Thomas was fine to continue, losing no time as the incidents occurred inside the final three kilometres, while Chris Froome managed to skirt around it.
"I'm alright," Thomas said.
"It was just a crash in front of me and I had nowhere to go.
"Luckily I took off most of the speed. I think Froomey managed to get round it and he's okay."
Thomas continues to lead from Froome by 12 seconds, with Michael Matthews of Team Sunweb on the same time in third place overall going into Wednesday's stage to La Planche des Belles Filles, where the fight for yellow is expected to ignite.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies