The 37-year-old Swiss star, attempting to triumph at the O2 Arena for the seventh time and the first since 2011, was beaten in straight sets by the Japanese world number nine.
Federer had won his previous 15 opening matches at the season-ending event, but put in an error-strewn display in his first round-robin match at London's 02 arena.
Nishikori had lost his previous six matches against Federer but, after winning a first-set tie-break, kept his nerve to rack up a surprise but thoroughly-deserved 7-6 (4) 6-4 victory.
"I'm glad to win, it's never easy to play with my idol," said Nishikori.
"It's always a big challenge to play Roger. The first match is not easy, both of us were playing a bit out, but the second set was better.
"I lost to him twice in the last two months so I played more aggressively and things started working, especially in the second set. There were some lucky points but I played well today."
Federer made an uncharacteristic 36 unforced errors, 20 in the first set, but that was to take nothing away from Nishikori.
Having feathered an outrageous backhand volley over the net to help take the first set to a tie-break, Nishikori then converted the fourth of five set points to take the lead.
Federer looked like he meant business when he broke Nishikori in the first game of the second set, but then he handed the initiative straight back with a desperately-loose service game.
Instead it was Nishikori who broke again to lead 4-2 and - after Federer forced him to serve for the match - the 28-year-old obliged for a memorable victory.
Federer admitted his practice has been "a bit all over the place" but had no excuses.
"I felt we both struggled throughout the first set. You could tell it was sort of a first round," he said.
"I had my chances maybe a bit more than he did. Then I started to feel better in the second set. I think we both did. The level went up but unfortunately I couldn't keep the lead that I got early."
Federer does, of course, still have the chance to progress in the tournament.
However, he will be wary of a big obstacle in his group in the shape of Kevin Anderson, the 6ft 8in South African who knocked him out at Wimbledon this year.
Anderson, who went on to reach the final at SW19 - via his epic six-hour-plus semi-final win over John Isner - is making his first appearance at the ATP Finals at the age of 32.
He looked determined to make up for lost time as he beat Austrian Dominic Thiem in straight sets, 6-3 6-6 (10)
French Open finalist Thiem simply could not make any inroads on the Anderson serve and admitted he lost to a potential title challenger.
The 25-year-old said: "He's very dangerous. He serves well. I think out of the taller guys, he's playing outstanding from the baseline. Of course, he can win the title here."
World number one Novak Djokovic opens his campaign against Isner on Monday evening.
Britain's Jamie Murray and Brazilian partner Bruno Soares won their opening doubles round-robin match, beating South Africa's Raven Klaasen and Michael Venus of New Zealand 7-6 (5) 4-6 10-5.
Register for free to continue reading
Registration is a free and easy way to support our truly independent journalism
By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists
Already have an account? sign in
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies