The ruling will please campaigners who say authorities have given the singer special treatment. Malawi's government, which came under fire after Madonna adopted a 13-month-old Malawian boy, had said yesterday it would support a second adoption.
Court registrar Ken Manda told reporters Madonna's bid to adopt Mercy had been rejected because the star was not a resident of Malawi.
An Aids epidemic in the impoverished southern African country has orphaned more than one million children.
In her ruling, Judge Esimie Chombo warned against celebrity adoptions, saying they could lead to child trafficking.
"Anyone could come to Malawi and quickly arrange for an adoption that might have grave consequences on the very children that the law seeks to protect," she said.
Madonna's lawyer, Alan Chinula, said she would lodge an appeal with the Supreme Court today. Her London spokeswoman was not immediately available for comment.
Madonna has entertained millions around the world with sexy high-energy performances and songs like "Material Girl" and "Papa Don't Preach", and created controversies along the way.
In 1989, the video for "Like A Prayer", with its links between religion and eroticism, was condemned by the Vatican and caused Pepsi-Cola to cancel a sponsorship deal with the star.
Madonna surprised fans in February by dedicating another of her hits, "Like a Virgin", to the Pope at a concert in Rome.
Malawian rights groups, who accused the government of skirting residency laws when Madonna adopted David Banda in 2006, also opposed the latest adoption attempt.
Prominent rights activist Mavuto Bamusi defined said the decision was a "defining moment for child protection" but urged the pop star to continue her charity work.
"We finally appeal to Madonna to take this positively and continue to help the children of this country," he told Reuters.
Malawian Information Minister Patricia Kaliati said yesterday that Madonna had helped in the country and was a worthy mother who was supporting over 25,000 orphans here.
Madonna's charity, Raising Malawi, plans to build a multi-million dollar school for girls in Chikhota village, about seven miles outside Lilongwe.
Madonna, accompanied by David, arrived in Malawi on Sunday ahead of the court examination of her application.
The star, who was divorced last year from British film director Guy Ritchie, is one of the most successful singers of all time, with album sales of more than 200 million.