His eldest son, Dai Llewellyn, said: "He had a long illness and finally today he died. He will join my mother `Teeny.' We are all very devastated, especially since we lost Mum, but he was miserable in the last couple of months and he will be glad to join my mother. They will be very happy together."
Dai, who inherits his father's title, said the family, including his brother Roddy and sister Anna, were all present when Sir Harry died.
Sir Harry, best known for his gold medal win at the 1952 Olympic Games on his beloved horse Foxhunter, is to be cremated on Friday and his ashes scattered on the mountain overlooking Abergavenny, where his famous horse was buried.
He and Foxhunter were feted nationally for winning the Gold Medal with the British equestrian team, including Wilfred White and Douglas Stewart, at Helsinki with a final clear round. He had earlier won a bronze medal, also on Foxhunter, at the 1948 Olympics in London.
Born in 1911 in Aberdare, and named Henry Morton Llewellyn, Sir Harry was the second son of colliery owner Sir David Llewellyn. He was educated at Oundle School and Trinity College, Cambridge.
Sir Harry worked hard over the years to become a senior figure in the equestrian world, where his close colleagues included the Duke of Edinburgh.
He was a regular steeplechaser before the war, being placed second and fourth in the Grand National in 1936 and 1937, respectively.
His military achievements were also recognised as he quickly rose in rank to Lieutenant Colonel Llewellyn and he was awarded the OBE in 1944.
Sir Harry worked as senior liaison officer to Montgomery for the whole of the north-western Europe campaign, earning the American Legion of Merit for his services.
Between 1971 and 1981 he was also the chairman of the Sports Council of Wales.