Corridors of power: Albany in central London has been a secretive refuge of the elite - until now

Home to prime ministers, film stars and poets, one block of flats in central London has been a refuge of the elite for more than 200 years. Albany resident, the antiques dealer Christopher Gibbs, offers a rare guided tour.

The longstanding rules governing life at Albany – the stately Georgian pile situated surprisingly, if conveniently, in the heart of Piccadilly, which I have had the good fortune of calling home for half my life – are mercifully straightforward: no pets, no children, no whistling, no noise and absolutely no publicity.

Photographers have found themselves escorted from the premises, London's oldest block of flats and one of its most exclusive addresses, for snapping a frame of the interior courtyard.

Residents who are deemed indiscreet risk a ritual scourging by the trustees. So ingrained is the sense of decorum that even to utter a friendly hello to a neighbour as we pass on the stone stairs or the covered outdoor canopy, known as the Rope Walk, might be violating a taboo, as if a few words of small talk could break the spell of privacy we Albanians prize so deeply. (A nod suffices, a hat raised to a lady.)

While its inhabitants are notoriously tight-lipped about the place, it has been home to many storied residents: royals like Lord Snowdon (for a moment); intellectuals like Lord Byron, Isaiah Berlin and Aldous Huxley (and the writer Sybille Bedford, who lived for a while in his servants' room); numerous politicians; and the crème de la crème of the style world, like the decorator David Hicks, Baroness Pauline de Rothschild, the American diplomat's wife Evangeline Bruce and the editor of Flair, Fleur Cowles.

Think of a monastery, but one in which the customary Trinity has been replaced by secular devotions – exacting taste, the pleasures of life and a romantic nostalgia for England's past – and you have a good feel for the place.

Location accounts for a good bit of Albany's appeal. Next door to Burlington House, home of the Royal Academy of Arts and other learned societies, a stone's throw from the enticements of Soho, the grandeur of St James's, and the comforts of Mayfair, to say nothing of the canny tailoring of Savile Row, lies this hidden world – part club, part cloister – stretching the full length of its neighbouring Sackville Street. "It's incredible to have such a tranquil haven in the dead heart of London," observes the esteemed art historian John Richardson, who lived at Albany from the 1960s through much of the 1980s. "It was sheer heaven."

Even its name has been the subject of excessive debate over the years: most initiates insist, as do I, on omitting the definite article, calling the place simply Albany, though Oscar Wilde, in The Importance of Being Earnest, and Charles Dickens, in Our Mutual Friend, saw fit to use "The Albany". To each his own! Likewise, its units are not referred to as apartments or flats but as "sets".

Precious few of the passers-by who throng Piccadilly each day have the slightest notion what lies beyond the building's imposing pedimented façade, ably guarded by a formidable team of liveried porters. Which is exactly how we like it.

While the doors have opened a crack in recent years, with the occasional set being snapped up by the odd hedge funder on the open market rather than passing quietly to someone in the know, even a resident of some 40 years, like myself, must be cautious about disclosure. For instance, even with a wonderful new book out to sell, David Hicks's widow, Lady Pamela (Mountbatten) Hicks, viewed the prospect of talking about Albany as out of the question.

Until it isn't, of course. The secrets of the place – and there are many – have gradually tumbled forth over the years, beginning with those of its first inhabitants, Lord Melbourne and his wife, Elizabeth, for whom the main mansion was designed by Sir William Chambers, one of King George III's preferred architects, as a palatial town house completed in 1774. Both lord and lady enjoyed numerous extramarital affairs (Elizabeth had children by Lord Egremont and by the Prince of Wales, among other paramours, as well as at least one by her husband), creating a salacious mythology that, rightly or wrongly, persists to this day.

After he squandered much of his fortune, Melbourne exchanged his grand house in 1791 with that of the king's son, the Duke of York and Albany, who installed his Prussian wife and her menagerie of cats, dogs and monkeys. Alas, the Duke of York was as extravagant and dissolute as his predecessor, and he, too, was forced to sell.

In 1803, an imaginative young developer named Alexander Copland purchased the property. He and the architect, Henry Holland, divided the space into smaller chambers and added two additional buildings, creating a total of 69 sets, some of which were subsequently recombined.

The shrewd Copland had noted the growing need in London for small-scale residences within walking distance of St James's and its clubs and the Houses of Parliament, where a country gentleman with no desire for an elaborate dwelling in town could feel at home, with his own wine and coal cellars down below and a modest billet for a valet upstairs. Copland marketed the apartments exclusively to well-to-do, socially connected and unencumbered men. No women were permitted on the premises (at least not officially) until the 1880s.

In his 1848 novel, The Bachelor of the Albany, Marmion Wilard Savage described it as "the haunt of bachelors, or of married men who try to lead bachelors' lives – the dread of suspicious wives, the retreat of superannuated fops, the hospital for incurable oddities, a cluster of solitudes for social hermits, the home of homeless gentlemen … the place for the fashionable thrifty, the luxurious lonely and the modish morose."

Among their number, early on, was the rakish Lord Byron. His more like-minded neighbours included Matthew Lewis, the master of Gothic horror and, per one obituary writer, a "reckless defiler of the public mind," whose scandalous novel The Monk won the enthusiastic praise of the Marquis de Sade. Another resident was the foppish novelist and opium addict Edward Bulwer-Lytton (the man who, incidentally, coined the term "the great unwashed").

Other Albanians, some of whose association with the place is commemorated in a collection of plaques and busts adorning the mansion's central corridor, include a few prime ministers (Lamb, Gladstone, Heath and, for just a few days, Thatcher); writers like the playwright Terence Rattigan; the actor Terence Stamp (of the sapphire eyes and chiselled cheekbones); and my friend Bruce Chatwin, explorer extraordinaire, who lived in my attic with a Jacob chair from the Tuileries and the 18th-century bedsheets of the King of Tonga adorning the wall. The glamorous society hostess and publisher Fleur Cowles occupied a grand apartment the width of the mansion, where she entertained everyone from Lady Bird Johnson to Princess Grace.

Several fictional characters made their homes at Albany as well, including the gentleman jewel thief AJ Raffles and Ernest, the raffish alter ego of John Worthing in Wilde's The Importance of Being Earnest.

Albany also provided the setting for the assiduously covered dalliance of Alan Clark, the irreverent Tory MP and diarist, not only with the wife of a judge but with her two daughters as well. f

I must not speak of my neighbours now, on penalty of banishment, but on my staircase live a distinguished actor, a philosopher, the widow of a celebrated jeweller, three pretty girls from New York, the secretary to the trustees and myself.

I've lived here for more than half my life, having already been something of a regular before making the place my home, a not uncommon route since new residents must be approved by the trustees and the secretary, who orders the life and rhythm of our sanctuary. In this, he is aided by our largest leaseholder, the Cambridge college Peterhouse, due to the benevolence of William Stone, big-game hunter, botanist and world traveller, who lived in Albany from the 1890s till his death at 101, systematically buying up sets through troubled times. In his will he left all these to his old college, which now has the delicate task of maximising the value of Stone's gift while at the same time preserving the spirit, ethics and style of the building for its inhabitants.

I had my first glimpse of Albany before I was 20, as the guest of an Oxford friend who lived in his father's attic room. It seemed then, and seems still, perfection for a singleton, an oasis of serenity in the frenetic heart of west London.

A few years later, Harold Nicolson, the diplomat, MP and erudite man of letters, entertained me in his ground-floor rooms full of treasures. Not long after, I attended John Richardson's spirited gatherings, where the scholarly and high-minded collided with wilder, more exuberant friends in rooms filled with Picassos, Pre-Raphaelites and Roman busts. Richardson fondly recalls the uppity antics of Prime Minister Heath's security men, remembers the courtyard choked with the chauffeured cars of Mrs Cowles's glittering guests, and his own subtle, almost lethal handling of a secretary once reluctant to admit him to the sacred precincts.

Before giving up his set and settling permanently in New York, Richardson took on a few tenants, among them Bryan Ferry of Roxy Music, "a pop singer, but very gentlemanly," and for a month or so, Greta Garbo, whom he identified only as Miss Brown. "Even the maid, who was a movie buff, never knew it was her," he recalls.

My Albany initiation was long and rich and enjoyable, and I leapt at the chance of a roost of my own in this cloistered world. My first berth was as a lodger in the attic room of a diplomat friend who had married late. When the impending arrival of his first child prompted a move from L6 – infants being about as welcome at Albany as the onset of gout – I moved down to the rooms where I have lived ever since. (Lucky fellows like me, and perhaps a dozen others on old leases, have rents fixed every three years by the state. The less fortunate have to pay hugely increased rates, and a freehold set, one owned outright, could cost upward of $3 million.)

My friend, the diplomat, let me in on a secret. The rooms were once haunted by a previous incumbent, an alcoholic Welsh baronet who had reportedly drowned – indeed, boiled – in his overheated bath. His spirit seemed lodged in a dumbwaiter that rose to the attic kitchen, jamming the works. A Jesuit priest exorcised him, gently and efficiently.

With that, I had the place to myself and proceeded to make it my own in the accustomed style: sober mahogany, Oriental rugs, family pictures. I've continually reordered these rooms, distilling, paring, pruning, as chaste as my voluptuous nature allows.

So enraptured with my Albany life did I become that, reason deserting me (not for the first or last time) I gave up a spacious Bond Street gallery where I had my antiques shop and took one at the back of Albany, a small if elegant space with an entrance into the building, another on to the street and best of all a door into the underground passage – a quiet, invisible stroll home.

I love returning from Morocco, where I now spend much of my time, seeing the friendly porters at the door, hearing the clip-clop of feet on the Rope Walk, seeing the old-timers sitting in the little ivy-lined garden, sunning themselves by the little bronze statue of Antinous and the fountain, regularly invaded by the ducks from St James's Park – and enjoying sudden glimpses into other people's lives through uncurtained windows, the hours marked only by the clock chimes of St James's Church, accompanied by the more uptempo tinkling of the clock at Fortnum & Mason, our local grocer's.

At this point, when I am in London I can't imagine staying anywhere but Albany – all the more reason, perhaps, to clam up, maintain the longstanding omertà (or I should say discretion) that has held sway for centuries now, and count my blessings.