Interpol has issued a global security alert warning that the escapers - including men who had carried out attacks on the American warship USS Cole and the French oil tanker Limburg - were a "clear and present danger to all countries".
The jailbreak was seen as a significant propaganda coup for Islamist groups and a severe blow to Yemen's attempts to portray itself as a strong ally of the West in the "war on terror".
Guards at the high-security prison, which also served as the military intelligence headquarters, were being questioned by police.
There is widespread suspicion that the audacious escape could not have succeeded without the help of officials in Yemen, the country Osama bin Laden's family came from.
The news of the escape came as 15 other alleged Islamists went on trial in Yemen on terrorism charges and as reports emerged of increasing presence of Islamist fighters in the region.
Yemen was long regarded as a haven for Muslim militants and had seen a series of abductions of foreigners by armed gangs. Following the attacks of 11 September 2001 in the US, facing an ultimatum from Washington, the Yemeni government announced it was cracking down on extremists and carried out a number of arrests.
Diplomats in the capital, Sanaa, spoke of "serious errors" that allowed the escape to take place and the probable culpability of Yemeni officials. Opposition parties blamed the government. Ali al-Safari of the Communist Party said: "Nobody, not even al-Qa'ida, could have done this alone." Mohammed al-Sabri, of the Nasserite Party, added: "This is a serious setback and puts the country in a very embarrassing position. This has implications not just for Yemen, but all countries in the region."
Among the fugitives were Jamal al- Badawi, Fawaz al-Rabihi and Abu Assem al-Ahdal. Badawi was convicted of bombing the USS Colefive years ago, in which 17 sailors were killed.
Last year Badawi's death sentence was commuted to 15 years' imprisonment. This was his second jailbreak. He had escaped while awaiting trial in 2003 and was subsequently recaptured.
Rabihi was sentenced to death over the attack on the Limburg two years ago when a Bulgarian crew member was killed and 90,000 barrels of oil spilled into the Gulf of Aden. Ahdal is believed to be al-Qa'ida's deputy leader in Yemen.
The prisoners are thought to have made their way out of the mosque on Thursday night and then used the cover of worshippers leaving after Friday prayers to disappear.
The secretary general of Interpol, Ronald Noble, asked the Yemen government to provide any relevant information to law enforcement agencies in other countries. He said that unless they were caught soon "they will be able to travel internationally and to ... engage in future terrorist activities".
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies