No Kenyan presidential election has ever been nullified and the ruling shocked a country that had been braced for further protests by opposition supporters.
"It's a very historic day for the people of Kenya and by extension the people of Africa," said opposition candidate Raila Odinga, who had challenged the vote.
"For the first time in the history of African democratisation, a ruling has been made by a court nullifying irregular election of a president. This is a precedent-setting ruling."
The six-judge bench ruled 4-2 in favour of the petition filed by Mr Odinga.
He has claimed that the electronic voting results were hacked into and manipulated in favour of Mr Kenyatta, who had won a second term with 54% of the vote.
"A declaration is hereby issued that the presidential election held on August 8 was not conducted in accordance to the constitution and applicable law, rendering the results invalid, null and void," Chief Justice David Maraga said.
The court did not place any blame on Mr Kenyatta or his party.
The lead counsel for the president, Ahmednassir Abdulahi, told the court that the nullification was a "very political decision" but said they will live with the consequences.
Mr Odinga's lawyer had asked the court to invalidate Mr Kenyatta's win, saying a scrutiny of the forms used to tally the votes had anomalies that affected nearly five million votes.
Kenya's electoral commission had said there was a hacking attempt but it failed.
International election observers had said they saw no interference with the vote.
Mr Odinga, a longtime opposition candidate and the son of Kenya's first vice president, had unsuccessfully challenged the results of the 2013 vote.
His supporters at first had said they would not go to court again this time but filed a petition two weeks ago.
Kenya had been braced for further protests on Friday as the court prepared to rule on the opposition's challenge, with police deployed to sensitive areas of the capital Nairobi.
Security was tight around the courthouse, with armed police and barricaded streets.
Human rights groups have said police killed at least 24 people in unrest that followed the August 8 vote.
Many shops in the capital remained closed.
Unease around the election rose when the official who oversaw the electronic voting system was found tortured and killed days before the vote.
But the unrest following last month's election was far calmer than the post-election violence a decade ago that left more than 1,000 people dead.
Opposition members danced in the streets, revelling in the setback for Mr Kenyatta, the son of the country's first president, in the long rivalry between Kenya's leading political families.
"This has shown all (election) observers did not do their job. We want an apology," said John Wekesa, who was dancing outside the court.
The court said the election commission "committed illegalities and irregularities ... in the transmission of results, substance of which will be given in the detailed judgment of the court" that will be published within 21 days.
Mr Odinga called for the election commission to be disbanded and said the opposition will ask that electoral officials be prosecuted.
Electoral commission lawyer Paul Muite said: "Right or wrong, the Supreme Court has spoken.
"So what remains is a fresh opportunity for the people of Kenya, in exercise of their sovereign authority, to once again restate with clarity who they want as their president".
Register for free to continue reading
Registration is a free and easy way to support our truly independent journalism
By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists
Already have an account? sign in
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies