The official from Portland, Dorset, has since seen the incident on television in which Le Saux appeared to elbow Fowler on the back of the head and claims he would have shown the Chelsea and England defender the red card.
"I can assure everyone if I had seen it Le Saux would have been sent off," he said. "But both myself and my assistant referee did not see it because if was off the ball so, no, it will not be in my match report.
"I saw it on television this morning and it's clear for everyone to see something has happened.
"What happens now is down to the FA, but from a personal point of view it was just a little bit annoying that I missed the incident."
The FA has used television evidence against players before, such as when John Hartson kicked Eyal Berkovic earlier this season, and Le Saux could face disciplinary action even though Durkin did not see the incident.
The BBC will provide the FA with a tape of the incident if asked and an FA spokesman, Steve Double, said: "It is an option while there is also the fourth official's match report to consider."
The incident was the culmination of a running battle between Le Saux and the Liverpool striker, for which both players were booked.
The feuding pair were highlighted by Trevor Brooking on the BBC's Match of the Day and he felt Le Saux and Fowler had been lucky to escape red cards for their antics.
"I think both players were fortunate that they got away with just yellow cards each," he said.
"There had been a lot of verbal goadings going on from Robbie to Graeme, because initially Robbie felt Graeme had made a little bit of a meal of a tackle down the line when he had stayed down. Robbie was booked for that and it took a long time for Graeme to take the free-kick and he got booked as a result of not taking the free-kick.
"They carried on for the next two or three minutes until the incident and it did appear from the video evidence that there was an elbow to the back of Robbie's head.
"That was not spotted by the referee while the linesman's view was blocked by another Chelsea player.
"They did not know what had happened and they just saw Robbie was down so when Paul Durkin called them over, having not seen the incident he could not react to it."
Le Saux has gained something of a reputation for indiscipline over the years and earlier this season he was sent off for punching Blackburn's Sebastien Perez, while in 1995 there was his infamous brawl with his former Rovers team-mate David Batty.
On that occasion, he was involved in an onfield fracas with Batty during Blackburn's 3-0 Champions' League defeat against Spartak Moscow in Russia. The two players squared up to each other and appeared to exchange blows.
In December 1997, after he had rejoined his first club, Chelsea, following his spell with Blackburn, Tottenham fans accused Le Saux of making rude gestures during the Stamford Bridge side's 6-1 win at White Hart Lane.Reuse content