Greed does have a face: it's called Olympic Bid City. Cape Town is full of estate agents matadoring around with offers to purchase. In a flanking manoeuvre. with diversionary thrusts and pincer movements, real estate here doubles in price overnight. Cape Town was once so unknown it didn't even appear on weather charts at airports. Nobody, unless they were bats or had a grandfather mentioned in despatches in the Boer War, had ever gone into a travel agency and said, "I want a ticket to Cape Town."

However, we were never quite alone. Odd people fetched up on our shores: a tonto Italian who had travelled the world on a Vespa and sucked petrol from passing cars; a Fellow of the Finger Print Society; widows who came out on the Union Castle Line and stayed at the Mount Nelson - divinely faded in those days, with ancient waiters who shared one set of false teeth.

There was the famous Villa Set, an often fatally flawed group, including a man called Bungalow Bill and an ex-Dragoon Guard who slipped into a crimson crepe kimono and high-heeled cork sandals after dark. They all lived by Hilaire Belloc's "The nuisance of the tropics is the sheer necessity of fizz".

The town attracted people who liked to live above the cake-line. Patricia O'Neill, daughter of Lady Edith Kenmare, who had a penchant for men with names like "Caviar" Cavendish, came from Kenya and set up house with a baboon called Buster, who watched videos with his nanny (an ex-policewoman from Bristol) and munched through Patricia O'Neill's Cartier pearls. O'Neill had been married to an Australian Olympic swimmer, and on their wedding night, on a yacht moored at Cap Ferrat, he dived overboard and swam away.

In the Eighties, the Aspinalls bought a house, with so much sea view you could drown, for the same price as a flat in Earl's Court. They imported mattresses from Heal's and sucked up to wild animals. Lady Sarah carried orphan chimps from Rwanda, wrapped, in swaddling clothes onto aeroplanes. Billy and the Kid arrived with six containers of designer clothes. They had a dinner party and one of the guests said, "What position does Jean Paul Gaultier play again?" He thought he was a rugby player. Billy cried.

Then there is Sir William Piggott Brown, who lost millions in one night of gambling and now lives on Clifton Third Beach and makes a few urgent calls round about the time of the Maiden Plate.

The Villa Set didn't care about much, including apartheid. A grand member now says peevishly, "Why didn't someone tell me about it?" While they became serene masters of relaxation, the place filled up with a lot of immigrants who looked like Ronnie Biggs, knew about close quarter combat, and had wives with pizza knees. This was the era of Holly and Roger Hamilton Brown. She was adorably pretty and wore skirts as brief as a haiku that revealed a flick of white knicker when she crossed her legs. He was once the Flaming Viking Stripper in Bromley but, when he tried to recreate the act for his fortieth birthday, he nearly burned to death. Somehow, he'd lost his snap. They were in every magazine, standing in front of a bit of uber-decorating or slowly turning the colour of tandoori chicken beside the swimming pool. When she left Cape Town, Holly said, "Great scenery. Pity about the neighbours." She, like many others, had tapped its parochial heart and perceived social treachery and superficiality. The line is "World- famous in Cape Town", and the irony is that when Mark Thatcher was blackballed by a local golf club, it was a boggling act of pettiness entirely in keeping with the town's tin heart.

During the years of apartheid, the foreigners who came were foreign correspondents. They famously preferred The Mount Nelson to the Dolly Hotel, Khartoum. The Mount Nelson had sheets. Many of us were led to swimming pools (shorthand for Rich and Vulgar) bravely holding drinks with little umbrellas in them. The footage dissolved from pool to shack-land and a commentator - frequently the censorious face of Michael Buerke - saying, "While Tracey (every female in South Africa is a Tracey) worries about putting on weight, her domestic help goes home to a cardboard box without electricity or water."

The people who now arrive are too grand or too awful to mix much with locals. I used to go out occasionally - a girl likes a bit of fun - but if Earl Spencer is invited, journalists are not welcome. A girl from the Daily Express once rang me for stories; she had a particular way of purring the word "pounds" that was quite catchy. I mentioned "Ten Dead in Taxi War". She mentioned Charles Spencer.

Cape Town has always been a chic-free zone. Hail to the last suburb in Africa, Windsor Close - kidney-shaped swimming pools, bathrooms en suite, and rotary clotheslines. A friend abroad says she misses the sound of sprinklers on the lawn. By the time you read this, we should know about the Olympics, but, even if we don't get it, our cover is blown. We're alone no more - brought from glorious isolation to the world of backpackers and Scandinavians having unprotected sex in cheap hotels