Morgan McSweeney will instead take up a "strategic role" within the leader's office, according to the party.
"Morgan remains Keir's number one adviser. He is staying on, staying in the top team and staying in LOTO (the Leader of the Opposition's Office) focusing on getting Labour election-ready for 2023," the source said.
It follows the departure of communications director Ben Nunn and deputy communications director Paul Ovenden within hours of the Chesham and Amersham result.
Concerns are now growing over Labour's prospects in the by-election in Batley and Spen, which was murdered MP Jo Cox's seat, on 1 July.
A new poll put Ms Cox's sister Kim Leadbetter on course to lose the Yorkshire constituency to the Conservatives by four per cent of the vote at 47 to 43.
The by-election was triggered after Tracy Brabin, who won the seat for Labour in 2019 with a 3,525 majority over the Conservatives, was elected as the mayor of West Yorkshire in May.
Labour came fourth in the Chesham and Amersham by-election with just 622 votes, or 1.6 per cent of the votes - meaning it lost its £500 deposit.
Polling expert Sir John Curtice said it was the "worst Labour performance in any by-election".
Mr Starmer reshuffled his shadow cabinet after several disappointing results during the May elections, and is now overseeing a shake-up of his staff.
In Mr Nunn's departure email to staff, the aide said he was standing down from his role to "go forward and do something different", but claimed that he believed Mr Starmer "will be a great prime minister".
By contrast Mr Ovenden left his job for family reasons. "It’s been an incredibly tough decision to step back from a job I love but at this difficult time, family comes first," he tweeted. "I’ll be back digging up stories before you know it."
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies