Lord Faringdon faced the verbal attack moments after an industrial tribunal ruled that he had unfairly dismissed Tim Clarke, who was not awarded compensation.
Mr Clarke's wife Judy, 51, lost her claim for constructive dismissal against Lord and Lady Faringdon.
In the witness box Lord Faringdon, 58, defended their treatment of Mr and Mrs Clarke.
Mr Clarke was employed as a janitor at Buscot Park, near Faringdon, Oxfordshire, while his wife was promoted to temporary housekeeper after two years' service as a cleaner.
The tribunal in Reading, Berkshire, heard that Mr Clarke lost his job after his wife had resigned in protest at not being given the full-time job of housekeeper, which was accepted by another woman.
Mr Clarke claimed the reason for his dismissal was that Lord and Lady Faringdon needed the grace-and-favour flat the couple shared at the National Trust home.
In angry clashes at the tribunal Mr Clarke, 46, was accused by Lord Faringdon of failing to do his pounds 10,000-a-year job. Lord Faringdon claimed windows were left unwashed, there was dirty water in the swimming pool and a squash court was left unbrushed.
As tribunal chairman, John Hollow, said he was not going to award compensation, Mr Clarke, turned to Lord Faringdon and said: "I hope you die."Reuse content