Ashley, the British and European light-heavyweight champion, would have picked up $100,000 (pounds 62,000) for fighting the former six-weight world champion on 7 June, but Emanuel Steward, the American's former manager, advised him against taking the bout.
Hearns is now self-managed but still turns to Steward, whose training methods have revived the career of the World Boxing Council heavyweight champion Lennox Lewis, for advice.
"Manny knows all about Crawford and he told Hearns he was too dangerous to tangle with," Frank Maloney, Ashley's promoter, said.
Now the Leeds-based Ashley will make a much shorter trip to Alfortville, outside Paris, to defend his European title against the Frenchman Pascal Warusfel on Saturday.
Hearns is not the only fighter to turn down Ashley. The Welshman Nicky Piper, whom Ashley beat for the British title in November 1994, has declined to meet him for the European championship, while Mark Prince, from London, has forfeited his right to challenge for the British crown.
Despite being unbeaten for the last two years, Ashley is no longer rated in the top 12 by the World Boxing Organisation. Ashley's No 2 ranking has gone to the Dutchman, Eddy Smulders, who gave up the European title rather than fight the Yorkshireman. To add insult to injury, Piper is the No 1, with Prince in third spot.
The WBO's somewhat illogical findings have prompted Maloney to fax their championships committee, asking them to re-assess the situation, especially as, since the last ratings were published, Ashley knocked out the Spaniard, Roberto Dominguez, in three rounds for the European title.
A nonplussed Ashley said: "Boxing politics have denied me a world-title chance. Boxers should prove themselves in the ring. Now no one wants to fight anyone in the top 10."
Maloney added: "Crawford is not getting any younger - he was 33 last week - but he is getting frozen out. We even offered Chris Eubank pounds 350,000 to fight him, but he turned us down.
"If we keep getting rejected, we will consider moving Crawford up to heavyweight to fight for the vacant British title."Reuse content