The sisters, who won gold medals in 2000, 2008 and 2012, were beaten 6-3 6-4 by Czech pair Barbora Strycova and Lucie Safarova in a big upset.
Venus finds herself out of the tournament after only two days, unless she decides to play mixed doubles, having lost to Kirsten Flipkens in the opening round of the singles on Saturday night.
The 36-year-old did not speak to the press after that loss, with coach Mary Joe Fernandez revealing Venus was suffering with illness.
She was clearly below her best again in the doubles and Serena, who won her opening singles earlier on Sunday, could not find a high enough level to make up for it.
The world number one said: "It was a really tough match for us, a really tough draw, but we're used to tough draws. We didn't play our best whatsoever tonight. We played terrible and it showed in the result."
For Serena, it was a first ever loss in Olympic doubles. She missed out on Athens in 2004, when Venus suffered a first-round defeat alongside Chanda Rubin.
Both sisters hold four gold medals, making them the most successful Olympic tennis players of all time, and they are rare among their peers in how much they prioritise the Games.
This surely will be an Olympic swansong for both, but Serena, who won double gold in London, insisted the result was not too much of a blow.
"I wouldn't say it's devastating," she said. "We had a chance to compete for our country and we did the best that we can.
"We had a blast out there. It was a lot of fun and we'll always remember these moments and these matches, and I think that's what matters most.
"We've had so many Olympic doubles golds, so many grand slam championships. It's been unbelievable."
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies