A charmed life, in many ways, blessed with money, looks, connections. Yet so often cursed, or so it appears from the outside.
Last week in St Lucia, where he now lives, we walked round the Jalousie Plantation, his recent and very wonderful inspiration, a holiday resort in one of the most sensational situations in all the West Indies. Feel the quality. Wonder at the cost - around dollars 50m (pounds 33m). Count the staff - 375, including 52 security people, to cater for 115 bedrooms. Admire the effort that must have gone into its creation.
Look at the man himself. Aged 66, yet not a wrinkle. Little hair, that's true, but then he usually wears an amusing straw hat. Tanned, lean, fit, piercing blue eyes. He doesn't smoke, take coffee or tea, and rarely drinks alcohol. A perfect gentleman, charming, cultivated, at ease with all classes, all races. Still a close friend of Princess Margaret. 'She is one of the friends who has always stood by me,' he said. Hmm. What can that mean?
Then we strolled next door to his own house. Not on the Jalousie Plantation. He pointed out a wall they had put up and the planks used to barricade him in. What is going on?
He lives alone in a simple wooden shack, attractive in the Caribbean style, but so far without any water. He sleeps on a foam rubber mattress on the floor. In the morning, to wash his teeth and face, he sneaks next door into Jalousie and uses their lavatories. How can someone born with so much, who has created so much, be living like this? Ah, if only answers were as simple as questions.
We'll go back 200 years, to the founding in Scotland of the family's fortune, based on bleach, not beer as many people assume (that's Tennent). 'My great, great, great- grandfather brought the industrial revolution to Scotland.' By 1925 they had sold out the messy, manufacturing side of things, by which time the family were landed gentlemen connected to Asquiths, Wyndhams, Lytteltons, and aristocrats in their own right. The first Baron Glenconner was created in 1911.
Colin was born in 1926, went to Eton and Oxford. In 1953, partly to get away from his family, he moved to the West Indies for a year to look at some Tennant land in Trinidad, 15,000 acres, which the Tennants had ignored for ages. 'It was like a graveyard, run down and badly managed. The West Indies was very mouldy in those days. No tourism, no airports. For 100 years these British-owned plantations had been run by people who were drunk or mad, mountebanks or refugees from justice.'
He came back to England, married Lady Anne Coke, daughter of the Earl of Leicester, then returned to the West Indies, honeymooning in Cuba, though his wife never liked the Caribbean, finding it horribly sweaty. The Trinidad land was sold and in 1958 Colin bought the island of Mustique for pounds 45,000. Sounds a bargain. 'Not quite. It was 1,400 acres of nothing. Not like buying 1,400 acres of Gloucestershire. I bought it for its beaches. I thought if everyone in the rest of the world goes on strike, I'll have this island.'
He returned to London, settled down to work in the family's offices in the City. When in 1963 his father decided to sell up completely, Colin was deprived of the firm he expected to spend his life running - but it turned out to be in his name. He ended up with one million pounds.
'Every man jack today is a millionaire, but in 1963 a million was worth something. I was frightfully rich. My father was never interested in money, but I always have been. Such a shame I have not been terribly successful at handling it. If I had done nothing with that pounds 1m, I would be very rich today . . .'
He decided to develop his empty island of Mustique, moving out there with his wife and children, building his own house, then building others for the rich and famous: fantasy houses, costing fortunes. Mick Jagger had one in the Japanese style. Bowie's was Balinese. He gave a plot to Princess Margaret for nothing.
There always seemed to be money coming in, from selling new sites and homes. He was virtually King of Mustique, responsible for health, roads, administration. The locals wanted schools for their children. The wealthy foreign home owners wanted a proper airport. Pop stars wanted modern telephones. All of which he had to install.
'It was frightfully successful, but then in 1975 everything started to go wrong. There was a high dollar premium, a crisis in secondary banking, wages, water and fuel rocketed. When I sold the houses, I had not built in an agreement for service charges. That was my mistake. People refused to pay and it all became very tiresome. I had to borrow money and sell my paintings.'
In the end, he was forced out, selling his stake for pounds 1m - equal, he says, to the total amount he'd put in. 'Now I should think it's worth about pounds 100m. I took no income in all the years I was there. It was very cruel.' Yes, but whose fault? 'My own. You should never sell to the rich. They always make sure they get the best value. The owners and bankers made all the money. I got a lot of publicity.
'I am capable of making money. I have good ideas. But I need a real brute to help me, keep an eye on me, someone hard and horrible. I didn't keep track of what I was spending, till it was too late. There was no tax to pay in those days, so I got into the habit of not keeping accounts.'
In 1979 he left Mustique and came to St Lucia. He bought a plantation, inland, planning to grow fruit and vegetables for export, but this did badly and he sold it. Then one day his son Henry went walking in the south of the island and discovered the Jalousie estate, virgin land between the Pitons. These are two mountains which rise straight from the sea, St Lucia's best-known landscape. Colin bought the land, paying pounds 200,000 for 488 acres.
Then began a struggle on two fronts - to get planning permission and get development money. This lasted eight years, during which time he lived in a fishing village in rented accommodation, often in some squalor. Several times he had all his possessions stolen, including his clothes. 'I took to wearing Indian clothes, so no one would steal them.'
There was a campaign against any development around the Pitons by those who consider it a sacred area. Derek Walcott, the Nobel Prize winner, who was born in St Lucia, was one of those who said it should all be kept a national park.
Planning permission was eventually granted, as long as the rainforest slopes were preserved, and after endless meetings round the world, the money came from an Iranian family. He sold them half the land, plus the planning permission, for a sum he won't reveal. 'I made no profit. I spent about dollars 100,000 a year for those eight years, setting up the project. Since 1966, I've had no income.'
He now owns no part of the Jalousie resort, despite what most people in St Lucia believe. He is clearly not on the best of terms with the Iranians, though the American company which manages the hotel appears friendlier.
'I don't want to talk about the relationship with Lord Glenconner,' said the general manager, Bill Banmiller. 'There is a perception that he is still a major player in Jalousie, which is not true, but he has done great service with his family name.'
Half the hotel's guests come from Britain. When they see Lord Glenconner strolling elegantly around, many imagine it is his hotel. The main reception room is called the Lord's Room and contains his family furniture, plus a table made by Lord Linley, but these are just on loan, till he has his own place again.
'The more I look at my life, the more it seems to have been a complete waste of time,' he suddenly said. We'd had dinner and were now walking in the dark to his wooden house. So far, despite his money sagas, he'd been frightfully cheerful. Oh come on, you've lived well these last 30 years, and can take satisfaction from two beautiful creations.
'No, I should really have used my brain more, perhaps become an academic. I did after all get a scholarship to New College. Or I should have followed my interests, such as the arts, and not tried to make money. If I'd kept my paintings, I would have had status and a fortune today. I sold my Constable, Whitehall Stairs, for pounds 63,000. Probably worth pounds 25m today. I had 15 Lucien Freuds. They'd be giving me dinners at the Royal Academy if I had them today.'
I asked if he believed there was some sort of curse on the Tennants, thinking of his three sons. He dismissed the idea. He wouldn't talk about his sons, except to confirm that Charles (the ex-drug addict) is living in Edinburgh while Christopher is managing well since his accident. He also has twin daughters, born later - a complete surprise. One was due to visit him in a few days, though he still had not found a place for her to sleep. He comes home to London in the summer to join his wife.
'People say to me: 'Oh, how ghastly, having a son into drugs', but far worse things can happen. During the First World War, 13 out of 16 Tennant first cousins were killed. That was a real disaster for the family. Then think of Bosnia. None of our women has been raped or killed. You just have to bite on the bullet, and somehow survive.'
He doesn't plan to return permanently to Britain ever again. He finds us 'bitter and envious', preferring the West Indies, the climate, the attitude to life. 'I once took six West Indians to Ghana and life there is feudal. Compared with that, I think the slaves were lucky to be taken from Africa, in that their descendants have a better life here, with no civil wars.
'In the West Indies people are not neurotic, so they don't notice my neuroticism. They have a simple, fatalistic attitude to health and death. Last week there were three deaths by drowning on a beach not far from here but people accepted it without any fuss. Such a relief when people don't talk continually about their health, or their pills. I agree with what Lady Diana Cooper used to say: there are only three statements you should ever give out about your health - Comfortable, Sinking and Gone. That's all there is to say.'
Despite his own radiant health, he is going to refuse from now on to be photographed, unless with his wife and daughters. Yet in the old days he was forever posing. 'I've attracted publicity ever since my christening. Now I'm too old. All old people look pretty much the same. I don't want to draw attention to myself any more.'
He has, however, not completely lost his touch. In his house I noticed a photograph of some elephants which he has been sent on approval. For 10 years he had an elephant called Bupa, who died two months ago. 'People used to say how cruel, keeping a single elephant, she must be very lonely without a mate. People these days think about sex all the time. My elephant was quite happy just to stand there all day eating, and not being interfered with, but I think this time I will get a pair.'
That's going to cost you. 'I hope not. I'm negotiating with a magazine who will sponsor them.' Very smart.
So also is his next venture. Yes, you can't keep a good developer down. His little wooden house is to be the focal point of a settlement he is creating on his half of the Jalousie estate, using buildings transported from other parts of the island. The House of Flowers opens this summer, with a restaurant, bar and shops. Then he's going to sell off 45 plots for holiday homes. This time he's thinking small, and doing it by himself, with what's left of his own money.
'I try not to think of the past. It's been a sorry tale. What matters is the future. As a developer, I'm probably a late developer. And I have got staying power.'
Hunter Davies was highly commended in the Feature Writer category of the British Press Awards. This is the last in his present series of interviews - he will return in the autumn. Angela Lambert takes over next week.