The outlawed FIS leadership in Europe welcomed the move as a "positive act ... and a definite and effective contribution to solving the crisis and returning to peace and stability".
The military prosecutor's office in Blida, where Madani had been sentenced to 12 years' imprisonment in 1992, said the measure was a "release on parole".
"It's a dramatic move in terms of the government relationship with the opposition and FIS," said a senior diplomat in Algiers.
"But it is not going to stop the violence like the bomb in Baraki ... or the massacres. There's no reason to think that Madani can do that now."
The FIS leadership urged Algiers to follow Madani's release with that of his fiery deputy Ali Belhadj, jailed at the same time for 12 years, and that of "all the political prisoners".
It said this was needed to prepare the climate to "allow the return of the effective sovereignty of the Algerian people".
Belhadj is widely seen as a more of a hardline radical than his boss. The authorities, during their talks with the FIS leaders in 1994 to try to end the violence, accused him of sending a letter to guerrillas urging them to increase attacks.
Madani's release follows an upsurge of violence since Algeria held multi- party general elections on 5 June. Hundreds of people have been killed in attacks blamed on Muslim fundamentalists.
"And the GIA has nothing to do with the FIS and it's that group which is doing the urban bombings like that yesterday."
The Armed Islamic Group (GIA), led by Antar Zouabri, 27, is the most ruthless of the groups and is blamed by the authorities for bombings and massacres in which hundreds have died.
On Monday, a bomb in the Algiers suburb of Baraki killed 26 people and wounded more than 80, mainly women and children, Algerian newspapers said yesterday. Security forces said 21 died.
After a weekend in which two newspapers reported the massacre of 44 people, most with their throats cut, the government paper El Moudjahid said security had worsened with another massacre - this time 15 civilians - in Medea overnight on Monday.
Madani, 66, was serving a 12-year jail term imposed in 1992 for threatening state security after a strike and riots preceded Algeria's multi-party general election planned for June 1991.
In the unrest at the height of the FIS's domination of city streets in June he said: "If the army does not withdraw we will be obliged to call a jihad (holy struggle)."
Two days later, he and Belhadj were arrested. The poll was put off to December 1991. And after the FIS took a huge lead, the authorities cancelled the results and a planned second round.
Just months later, amid growing violence, Madani and Belhadj were tried and sentenced by Blida military court.
Last week Abdelkader Hachani, the man who led the FIS in the elections, walked free from jail after being given a five year sentence, covered by his period in detention. The prosecutor asked for 10 years on charges of trying to undermine state security.
El Watan newspaper then forecast President Liamine Zeroual's government might lift the state of emergency to forge political consensus with the opposition.
Legal opposition groups have long demanded the 1992 state of emergency be lifted to enable full political activity by Algeria's 29 million-strong population.Reuse content