Mr Fraser thought he had struck a great deal when he sold his solid, respectable stamp dealership, Stanley Gibbons, for pounds 13.5m last April. But there was a problem, familiar to all stamp collectors. His newly acquired wealth was only on paper. It was held in the shares of Flying Flowers, the Jersey-based group that supplies flowers by post. Flying Flowers had bought Gibbons amid spring talk of blossoming profits.
Sadly, by the summer, such hopes had faded. Sales of lobelias, petunias and begonias through the post were not doing well as the bedding plant market wilted. Flying Flowers shares slumped 45 per cent in a day after a profits warning.
"I was in Washington on holiday," says Mr Fraser this week as Flying Flowers announced their annual results, showing pre-tax profits well down at pounds 5.1m. "It was 8.12am on July 14 when the phone woke me and I was told about pounds 6m had been wiped off my shareholding. What did I do? I had the cheaper breakfast."
Suddenly a dream which he had spent nine years building (though he swears he does not collect stamps himself) had been shattered for him at the age of 43.
Shares which had been worth 555p each when Mr Fraser bought them in April were suddenly trading at around 300p. Nor can Mr Fraser's mood have been much improved by the knowledge that directors of Flying Flowers sold hundreds of thousands of shares at 550p just days after he had bought into the company. And the news got steadily worse. "There was another profits warning in August. My shares fell another three or four million. So by then I had lost pounds 10m." He would have been better off selling Stanley Gibbons for a few Penny Blacks. Little wonder that by September an increasingly desperate Mr Fraser joined Flying Flowers full-time and was appointed joint chief executive, alongside Tim Dunningham.
So what went wrong? You might think that one of the world's most famous stamp dealerships and a mail-order florists never really had a great future together. But Flying Flowers saw an advantage in two companies coming together which specialised in selling from catalogues. They also had a first-day cover business which married well with Gibbons. The real problem seems to have been more than the incompatibility of products.
A Jersey-based broker gave his version. "I reckon they were so busy with the Stanley Gibbons acquisition, they took their eyes off the ball," he says. "They had this big advertising campaign back in 1997 when their sales of bedding plants did very well. But they didn't keep a close enough check on what the market was telling them the following year. They were anticipating a similar uplift in demand and had already arranged to meet that demand by outsourcing supplies of plants. The growth never happened and they ended up having to destroy quite a few unwanted plants."
The broker absolved the management of anything worse than uncharacteristic incompetence. "Tim Dunningham is very straight. And he is usually very good at handling mail order lists. He knows how to experiment with them and use the information well. That is why I was surprised they cocked things up so badly."
It is an account which the management of Flying Flowers now largely accepts. Tim Dunningham referred this week to "naivety and undermanagement" of a firm that is now the Channel Island's largest non- financial company.
Mr Fraser was tactful about poor management that nearly took his shirt: "I think it was a problem in recognising the responses to advertising and in forecasts getting through to management. The controls that needed to be in place were not there."
Was he bitter about fellow directors who had cashed in many of their shares soon after he had acquired his? "No, it was my choice to get involved in this venture, so I can't argue about what happened. I think everyone was plain with me about the company. It has been another rich experience under my belt. Onwards and upwards, I feel very positive about what we are doing at the moment."
Nevertheless, a shake-up is taking place in the board room. The chairman, Walter Goldsmith, a former director-general of the Institute of Directors, is standing down in favour of Roger Norbury, former chairman of investment banking at NatWest Markets. Mr Goldsmith presided over a long expansion in Flying Flowers. But his record was somewhat blotted latterly by stating, just after the Stanley Gibbons acquisition, that it "would enhance shareholder value". Not, sadly, for Mr Fraser.
So what is the future now for his investment? The prospects for a big breakthrough in the bedding market do not look great. The Sunday supplements are full of advertisements from the big seed suppliers such as Cuthberts who have diversified their activities. Flying Flowers faces a more competitive market than in the past.
But Mr Fraser is hopeful. "All I can say is that the future looks good. Looking at television, for example, it is very encouraging to see how gardening is so high up on the list of interests." He may be banking on stamp collectors more than hanging baskets to make him back his millions, a judgement with which analysts would agree.
"We have a terrific brand name in Stanley Gibbons," he says. "It is the perfect product for the Internet. We have an awful lot of information on our site. After all we have been publishing catalogues for a hundred years, so we could easily become the main site for collectibles."
Like any wise collector used to keeping his wealth in bits of paper, he declines to be drawn on when he will be worth pounds 13.5m again. "I'm taking the long view," he says with a laugh.