Holy Picasso!: The Brit at the centre of a £6bn art row

Derek Gillman is an English academic in charge of the biggest private collection of art in America. He is also the man accused of evicting this £6bn treasure from its rightful home – and of betraying the radical vision of the man who created.

Rocking in a swing chair on his porch, Walter Herman sips iced tea. Retired, he exudes the implacable calm of the cardiologist he once was; his wife, Nancy, is more clearly exasperated. "We're not giving up," she says, with a sharp smile. "We've done everything we can in the courts, so it's time for a little guerrilla warfare. Dr Barnes used to leave notices in phone boxes, and we're going to do the same thing – leaflet the ordinary people of Philadelphia, the people Barnes meant his collection to be for." Her husband nods, then looks over his shoulder, across North Latch's Lane. "When I was young, I used to work as a lifeguard," he says. "I always used to marvel at how peacefully people could drown. They would be in real trouble, but there'd be no noise. Unless you were right next to them, you'd have no idea of their anguish. That's what it's been like at the Barnes." The Barnes. Across from the Hermans' is another, larger building, set back from the road. Properties are big in Merion, the oldest and richest of Philadelphia's western suburbs. The one opposite the Hermans' is at the end of a curved drive, on 12 acres of wooded land.

Built in a close-faced classical style, it might be the home of a successful businessman; indeed, it once was. The only thing to suggest that the building is different from others in the road are the carved stone reliefs over its front door, the work of the cubist sculptor Jacques Lipchitz. Even these do not prepare you for what you find inside 300 North Latch's Lane: the biggest private art collection in America, including 69 Cézannes – more than in all the museums of Paris – 181 Renoirs, 46 Picassos, 59 Matisses; superb collections of African sculpture and Pennsylvania German ironwork. All these were bought by Albert Barnes. It is his Foundation the Hermans have fought to save for more than a decade.

Like Walter Herman, Barnes was a physician – his fortune came from an eye-medicine called Argyrol, patented in 1902 when he was 30. Argyrol was reputed to prevent gonorrheal eye infections in infants, a problem brought back by servicemen returning from the First World War. Barnes's eye-drops made him rich, although their link to venereal disease and his birth in the slums of Philadelphia excluded him and his wife, Laura, from Merion's drawing rooms.

Snubbing Barnes was a dangerous thing to do. When he amassed his collection in the early 1920s – his chief adviser was Gertrude Stein's brother, Leo – it was partly as revenge. Although Barnes's collection was surrounded by the kind of well-heeled folk who might flock to see Cézannes, his neighbours were barred. For half a century, until the hot June day in 1951 when – typically shooting a red light – Barnes ran his car fatally into a lorry, his collection could be seen by appointment only. And appointments were granted to what Barnes called "the plain people", particularly if they were black.

A mythology has sprung up around Barnes's collection, fuelled by those many people who weren't granted appointments. TS Eliot was one, the Pulitzer Prize-winning novelist James Michener another. ("You lick [socialites'] boots ad nauseam and comfort the smug complacency of snobs," wrote a gleeful Barnes to Michener. The writer, unbloodied, snuck in disguised as a steel-worker.) The critic Alexander Woollcott got a telegraphic no to his request signed "Fidèle-de-Port-Manech". Fidèle was Barnes's dog.

Yet, for all his chippiness, Barnes was a cultivated man, reading avant-garde art theory in German and following the ideas of critics such as Roger Fry. His collection's annoyance value to Philadelphia society was merely the gilt on the gingerbread. Its real purpose was as a visual aid: the Barnes Foundation wasn't an art museum – it was (and is) an art school, teaching a critical method built on its founder's beliefs. (The Foundation, it should be said, has long since dropped its founder's highly discretionary admissions policy – although you still have to book in advance.)

Visit the Barnes and you will be struck by the oddity of its hang. Paintings sit in tight rectangles or "ensembles", perhaps five wide and three deep. These are arranged by size, form and colour – "rhythm" was the favoured word – rather than period or artist. A Cézanne with a road receding to the right may form one corner of an ensemble, a van Gogh with a road receding to the left another. Interspersed will be Pennsylvania Dutch ironmongery or carved spoons. This eccentricity has nothing to do with decoration. Encoded in Barnes's hang is a High Modernist message about the universality of the artistic impulse – a belief that all creativity comes from the same place. Barnes's ensembles tell us something we do not want to hear, which is that hierarchies – in art, but also of class or race – are illusions. Barnes may not have been the Bolshevik he joked he was, but he was genuinely anti-elitist. And it is that anti-elitism which now threatens his collection.

Across the road from the Hermans' porch sits Derek Gillman, their neighbour and nemesis. In his mid-fifties, Gillman is English. He is also a distinguished historian and educator; his last job was as head of Philadelphia's Academy of the Fine Arts. He sits in what was Laura and Albert Barnes's bedroom, now the office of their Foundation's president, a post Gillman has held for 18 months. With his dark brows and frameless glasses, he bears a distant resemblance to Giorgio de Chirico's portrait of Albert Barnes. Also like Barnes, Gillman is high-minded and strong-willed; in other circumstances, you can imagine the two as friends. Given that Gillman is here to close the Barnes down, however, this seems unlikely.

On 15 October, if the project he has been hired to oversee goes to plan, construction will begin on a new building for the Foundation. This will be five miles away from Latch's Lane, from the mansion Paul Cret built to house the collection in 1922. The new Foundation will be on a busy ring road called the Benjamin Franklin Parkway. The ostensible point of the move is that the Parkway is in downtown Philadelphia – but the road as it stands is unattractive and hard to get to by foot.

Like much of the project, the design of the new Barnes has been kept hidden. Despite the imminence of its construction, designs for the new building – by the firm of Tod Williams and Billie Tsien – are described by the Foundation's press officer as being "in the conceptual stage". It seems that rooms will echo those of the old Barnes in size and layout, and that the doctor's hang will be preserved.

One work that will apparently not make the move is Matisse's famous mural The Dance, commissioned by Albert Barnes for lunettes above the windows of his main gallery. After visiting Merion in 1930, Matisse described the Barnes as "the only sane place" for the display of art in America. When the canvases he shipped from France turned out to be the wrong size, he was forced to cut La Danse into bits and reassemble its parts. Matisse's late, great career as a maker of collages dates from that moment. The mural – perhaps his finest – will be replaced in the new Barnes by a copy.

Which begs an obvious question: why move the Barnes at all? Gillman is clear about this. First, there is the question of light. Other than in its top-lit upper floor, the Barnes is window-lit, the sunlight filtered through yellow blinds. As a result, Gillman says, "The colour balances are all off." Indeed, the syncopation of blues in Cézanne's great Bathers is lost in the current lighting, and other works are equally affected.

Then there is a question of numbers. Central to Barnes's teaching method was a belief that students should be taught in front of works of art: there are, intentionally, no classrooms at Merion. This, too, will change. Downtown, the galleries will be interspersed with classrooms. Gillman's hope is that this will allow student numbers to rise while providing an opportunity to leaven Barnes's 80-year-old method with a newer art history. (Pressed as to whether Barnes's ideas will continue to be taught, Gillman says, "Of course". When I ask how many of the current staff are Barnes graduates, he turns to his press officers and answers, "One".)

All this feeds the Hermans' fears. Like the rest of Merion, they are appalled at the thought of losing the Barnes, of its founder's aims possibly being swept aside. The couple run Friends of the Barnes, a vociferous group committed to keeping the Foundation where it is. The protest billboards along the Hermans' property are merely a sign of their intent. The Friends have taken the Foundation's board all the way to the Supreme Court of Pennsylvania to prove bad faith and mismanagement, and to halt the move.

In June, their case was finally dropped in the state's Orphans Court, the equivalent of our Chancery. Not, though, before the presiding judge, Stanley Ott, had noted that Pennsyl-vania's attorney general – required by law to defend the Foundation's interests – had "cheered on" the trustees, "preventing the court from seeking a balanced presentation". The attorney general's behaviour, said Judge Ott, "constituted an abdication of the office's responsibility".

Why the inappropriate cheering on? A recent study showed Philadelphia's population is shrinking faster than any other American city's. By moving the Barnes's $6bn worth of Cézannes and Renoirs to the Parkway, the state's governor, Ed Rendell, hopes to kick-start a "Museum Mile", swelling tourism revenues. US attorneys general are elected; breaking rank with state governors is not a good career move.

Then there are the foundations which have pledged $150m for the new Barnes, headed by the $5bn Pew Charitable Trusts. As a rule of thumb, bodies such as the Pew are keener on paying for new-build projects to which their names can be attached than on rescuing existing institutions. This is particularly true of the Barnes. "While I was at the Academy, I watched the Foundation spiral into the ground," recalls Derek Gillman, with a wince. "Its behaviour during the 1990s won no sympathy with the philanthropic community. And without that community, the Barnes could not survive."

He is referring to the unhappy story of the Barnes's then-president, Richard Glanton. Shortly before his death, Albert Barnes bequeathed the nomination of four of the five members of his Foundation's board to Lincoln University, a small, poor, all-black college near his country estate in Chester County. This bequest would come into effect on the death of Violette de Mazia, the Foundation's chief trustee. Since de Mazia – rumoured to have been Barnes's mistress – lived to 90, Lincoln took control of the board only in 1989. Glanton, a black Philadelphia lawyer, was named president that year.

What followed was a decade of lawsuits and counter-suits, in which its new director tried to break Barnes's indenture, taking its works on a world tour, trebling visitor numbers and selling artworks to pay for building repairs. Like other Merion residents, the Hermans were aghast at the plans. Growing the number of visitors meant turning Merion's quiet roads into a hell of tour buses. When the Barnes applied for permission to expand its car park, locals opposed it; Glanton responded by suing them for racism, a case dismissed as "frivolous". By the time he resigned from the board in 1997, Glanton had run through the Barnes's endowment. The Foundation was broke.

And so it hung in the balance until, in September 2002, the Barnes's new president, Bernard Watson, called Lincoln's president, Ivory Nelson, to say he was filing a petition in court to increase the number of Barnes board members from five to 15, wresting control of the Foundation from the university. (Julian Bond – chairman of the National Association for the Advancement of Coloured People and son of one of Albert Barnes's best friends – was appalled at the black college's treatment. "Your actions could easily be viewed as racially hostile," he wrote to the new board. They didn't reply.) If the perennially poor Lincoln agreed not to fight the move, Governor Rendell would arrange $80m in funding for the college. If not, Lincoln would face the might of not only Phila-delphia's political establishment, but also the Pew trusts.

In 2002, Pew's ambitious head, Rebecca Rimel, applied to the Internal Revenue Service (IRS) for leave to turn the private trust into a public charity. Although Rimel testified that the Barnes move had "no implications whatsoever" for her application, documents filed earlier with the IRS place the move at the centre of Pew's case – a fact unearthed by the Hermans, using the Freedom of Information Act. Rimel has subsequently been criticised for using the financial clout of one non-profit organisation to influence the future of another. But whatever the ethics of the case, the Barnes was leaving Merion and there was nothing Lincoln could do about it.

What exasperates the Hermans is the needlessness of it. "[Thomas Jefferson's estate] Monticello would get more visitors if it was nearer Washington DC," growls Walter. "Why not move that?" His wife cannot believe the new Foundation will be able both to handle a projected 200,000 visitors a year and continue teaching in the old way. Montgomery County has offered $50m to keep the Barnes where it is; Lower Merion Township has allowed an increase in visitor numbers to 475 a day, or 170,000 a year. Nancy Herman's plan would be to build a Barnes pavilion on Museum Mile – she suggests Frank Gehry as its architect – where visitors would be told the Barnes story before being bused the 15 minutes to Merion. The suggestion has been met with silence. The Hermans' suspicion (and they are not alone) is that the collection may be allowed quietly to fade into the Philadelphia Museum of Art's.

In his book The Idea of Cultural Heritage, Derek Gillman unpicks the link most of us make between art and place, between "movable chattels" and "architectural fabric". "We may choose to visit... Greek sculpture or [Chinese] painting in countries other than Greece or China," he observes. Although the book was written before he came to the Barnes, you can see how the ideas might apply. "The concept of sentimental wholeness is all very well," Gillman says, sitting in Barnes's bedroom under a Renoir. "But when it constrains action, you have to think again."

Of course, there's no rational answer to that. There is no way of quantifying the difference between looking at a mural by Matisse in a room where he himself stood and looking at a copy of that mural in a mock-up of the room. Yet accessibility isn't just about being downtown; it's about ordinary people being seduced by stories, by proximity to extraordinary men.

Horror at the move stretches far beyond Latch's Lane. To the New York Times's Nicolai Ouroussoff, it is "a crime", while the New Republic critic Jed Perl describes the pro-Parkway regime as "cultural commissars". Outside the offices of Philadelphia's great and good – the very people Barnes set up his Foundation to defend against – you will be hard pressed to find anyone who approves it. But the move will go ahead, and much will be lost.

For more information, visit www.barnesfoundation.org