Sir Ed said his decision was made to protest Mr Trump's perceived inaction on the escalating humanitarian crisis in Gaza.
He believes Mr Trump has the power to end the conflict and secure the release of hostages.
The move, which Sir Ed described as a way to send a message to both Mr Trump and Prime Minister Sir Keir Starmer, was criticised by the Tories as "deep disrespect" to the King.
Sir Ed said there was “no honour like an invitation from the King”, and he and his wife had prayed about the decision before making the announcement.