Paul Barker: GB: Gordon Brown's Great Britain

"Say not the struggle naught availeth." So wrote Arthur Hugh Clough, faced with the grim despondencies of mid-19th-century Britain. But sometimes those trying to raise the banner of social equality in early 21st-century Britain must sigh and feel they wish they had Clough's modest optimism to fall back on. While the day-to-day pressures of office demand prominence be given to this or that issue - and Islamic terrorism is casting a long shadow this weekend - there has been a consistent theme that has united the recently departed and the newly appointed prime ministers. Throughout 10 years of New Labour, both Tony Blair and Gordon Brown have steadily repeated their commitment to the gospel of equal opportunity. Yet the evidence - as David Willetts, the Conservatives' education spokesman, recently pointed out - is that social mobility has silted up.

In his first statement as Prime Minister, Gordon Brown spoke out with the true voice of the Scottish lad o' pairts: "I grew up in the town that I now represent in Parliament. I went to the local school. I wouldn't be standing here without the opportunities that I received there, and I want the best of chances for everyone. That is my mission..." Let no one murmur James Barrie's ungenerous comment that "there are few more impressive sights in the world than a Scotsman on the make". For all his pride in his own success, Brown would truly like to see something of that achievement extended to others

Related articles

• Sarah Sands: So the Home Secretary is a woman. That's no big deal
• A serving soldier writes to the new Prime Minister
• Brown's Wars Part 3: Overstretched in Helmand province
• Brown's Wars Part One: Under attack in the UK
• Brown's Wars Part 2: Meltdown on the frontline in Basra
• Leading article: Blair's incendiary legacy passes to Brown
• John Rentoul: We want change, but what we really want is more of the same
• Alan Watkins: A standing ovation? The House has taken leave of its senses
• The Columnists: 'A decent and clever Cabinet'
• Search the news archive for more stories

But in the past 10 years, notoriously, Brown's hand was at least as much on the domestic tiller as the nominal captain's. Efforts on equality of opportunity have been ill-rewarded. Once, the old Labour Party thought it had a magic formula in its drive to make all secondary schools comprehensive. Even the Conservative Party felt it couldn't deny that momentum. As Education Secretary in the 1970s, Margaret Thatcher herself approved more comprehensives than any such minister before or since. But the magic didn't work. The new ideas - beacon schools, city academies - arrived.

The unlovely social contrasts continue. In my native Pennine valleys, the Ridings School in Halifax, long condemned, is to be shut. It has served housing estates, on the bleak outskirts of the town, where no one who had any option wanted to live. The allocation policies for social housing focused on the nebulous concept of "need" as the only way to get such a home. The result was that social housing ceased to be a hoped-for ambition, which people looked forward to. It became a dumping ground for those who didn't, somehow, quite fit in. The concept of need destroyed the concept of community. In her vigorous recent polemic, Estates, Lynsey Hanley attacks the outcome: ghettos for those members of the working class, and the non-working class, who haven't made it. The biggest social division today is between two-earner households and non-earner households, often living on estates, where sometimes no one has ever received any money legally, except through state benefits.

The great snag about comprehensives was always geography. Parents, whether Labour or Tory, who had the money could gravitate towards the areas with the better schools. In London, I used to live just down the street from Camden School for Girls, a comprehensive where the Prime Minister's wife, Sarah Brown, was a sixth-former. When we came to sell, every person who looked round had young girls. The school catchment area was our unique selling proposition.

Desperately seeking a way out of this trap, local authorities - Brighton and Hove, for example - propose to select applicants by lottery. This is reminiscent of Gordon Brown's interminable fiddling with the welfare system. The former welfare reform minister Frank Field attacked all this fine-tuning in a study published last month under the title Welfare Isn't Working. Youth unemployment, Field noted, is rising. So is worklessness. And so, most damagingly, are the numbers of young people not in education, employment or training: more of them now then when Labour came to power in 1997.

All the talk about "social mobility" is, of course, partly a euphemistic way of discussing the perennial British - and especially English - obsession with class. Unfortunately, this obsession is as unlikely to disappear as French discussion of cuisine. The great English vice is not flagellation, despite the little cards glued up in central London phone booths. It is snobbery. And snobbery can dress in a thousand disguises.

Social mobility was driven forward in the later decades of the 20th century - let's say from the late 1950s through to the early 1990s - by a flurry of economic changes which were not specifically introduced with that social impact in mind. Heavy industries closed. In the mid-1970s, the traditional blue-collar families slipped into a minority in the electorate. (Look no further for the rationale behind New Labour's quest for new voters.) Other jobs arose, eventually in great profusion. But they were usually modest clerical or computer jobs. Amble around towns such as Swindon or Milton Keynes if you want to see this new class in its stamping grounds.

Not that it is a "new class", really. It's our old friend the lower middle class. People who are conscious that they've got on, and want even better for their children. These are the social and economic driving force in a post-industrial society. Does this make them any better loved? No. "Milton Keynes" has become a useful trigger for an easy laugh.

But no expansion goes on for ever. This one has already braked. It is slowed even further every time back-office functions are transferred from Britain to the Far East. On top of which you must factor in the unforeseen impact of another economic change: the movement of women into the workplace. Not many women could become coal miners or foundry hands, nor did they want to. Now, none of the newer job definitions excludes them. This is a good thing in itself, but it puts the men in a tougher world. And some lose out.

The iron law of unintended consequences prevails. If you speak to the admissions tutors at leading universities, they'll admit that wider entry for women students can mean replacing bright working-class boys from comprehensives with pleasant (and also bright) girls from private schools. Wimbledon fans, not Manchester City fans. Remember: the use of private schools has gone up, not down, under New Labour. Parents know where to buy a ticket to ride.

Many of the other engines of social mobility have proved to be as sputtery as the comprehensives. There is now a university - or should one say "university" - within spitting distance of almost every doorstep. But as higher education expanded, the old myth that all university degrees were roughly equal was shattered. London University's Institute of Education estimates that the initial average economic returns from a degree are very small. But an Oxbridge degree is not average, whereas a degree from the University of Mudshire is very average indeed.

The pecking order has not been conjured away. In a special report this year, The Economist pointed out, "As the financial value of some university degrees sinks towards zero, training as a plumber or an engineer will look more attractive." How very English it is to equate engineers with plumbers. Brunel would turn in his grave. No wonder we have so few major manufacturing companies led by engineers, rather than by accountants.

But is this the first glimmer of downward social mobility? Until now, the entire movement has been upward. But if the range of opportunities has stalled, what then? Many of the British, whatever their current status, seem to have prepared their default position: a majority still claim they are "working class". (In the US, the majority always claim to be "middle class".) "Clogs to clogs in three generations" was the old Yorkshire proverb.

In all the understandable concern for integrating and helping children of ethnic-minority backgrounds, policy-makers have taken their eyes off the main social failing in British schools. A Joseph Rowntree Foundation study, Tackling Low Educational Achievement, published on 22 June, found that "nearly half of all low achievers are white British males". Moreover, the white British - both girls and boys - "are more likely than other ethnic groups to persist in low achievement".

Children from ethnic minorities often come from homes with a high aspiration towards getting ahead. The streets of Southall, west London's "Little India", are lined not only with curry houses and sari shops, but also accountants, financial advisers, doctors and dentists. Where children come from homes where English is not the first language, this is "only a short-lived handicap", Professor Robert Cassen, who led the study, concludes. "African and Asian children commonly recover from it by secondary school." Which is encouraging, but not for that white majority.

Anyone teaching at an inner-city school will confirm the troubles they have with some white boys; also with some Caribbean boys who have, in half a century of settlement, absorbed many of the values of their white peers. It was long obvious that, in the fanatical concern for percentage differentials (did the average Bangladeshi boy, for example, do worse than the average white boy?), the sheer numbers were ignored. There are, and always will be, far more white children in Britain. Within that group lie all the big challenges for better social mobility.

The Cassen report reveals the hazards. All initiatives seem to flag, finally. Take poor reading and writing at primary school. The National Literacy Strategy worked for a while "but the results have plateaued", Professor Cassen reports. The Government has spent millions of pounds on its programme to combat truancy. But truancy figures have risen.

Gordon Brown calls out, time and again, for "change". "I am convinced," he said last week, "that there is no weakness in Britain today that cannot be overcome by the strengths of the British people." But almost all the efforts over the past 10 years have been generated by bright youngish men and women in and around Downing Street. Do they know the social score?

In the early 1950s, the sociologist and consumerist Michael Young left the central offices of the Labour Party, where he had helped to draft successful election manifestos, in despair at the party's top-down approach to government. He went out into the East End to find out what people's own values actually were. With Peter Willmott he wrote the classic Family and Kinship in East London, reissued this year by Penguin. We need to know what will really motivate the underperformers. The answer is unlikely to be Asbos all round. Young wrote: "The working classes have always suffered from forms of denigration idiosyncratic to each period of history." It's only the white working class that a Guardian columnist, or right-on Radio 4 comedian, dare mock.

The frustrations are intense. Those who would like to get out of those grim housing estates are finding it hard to find a house of their own to move into. Price are high; building rates are low. In walking around some of London's worst estates - including the Aylesbury Estate in south London, where Tony Blair press-launched his mission of social change all those years ago - I very seldom met anyone who objected to Mrs Thatcher's right-to-buy policy. After all, this was a greater transfer of wealth to the working classes than any step before or since. What they objected to was the lack of new, extra houses being built. Before long, you will need a bonus from Goldman Sachs to buy an inter-war semi.

A Yale University sociologist, Peter Marris, who died last week, described the great changes in access at Yale in his lifetime. Once, Yalies were "all men, mostly from well-to-do families, whose fathers had often been there before them". In the Yale Club in New York hang huge oil portraits of recent Yale presidents of the United States: both the Bushes, Ford and Clinton. Only Clinton, a man who clambered up from among poor whites, falls outside Marris's definition. But now half of Yale students are women. Some students have struggled up the Clinton route. Special admission rights for the children of alumni are being squeezed. New Labour, also, has tried to press universities to go this way, though very clunkingly in the case of Gordon Brown, in 2000, intervening in Oxford's rejection of Laura Spence with her 10 A-stars at GCSE.

But privilege always has another defence, Marris noted. Prosperous American parents push their children into summer maths programmes or latrine-building in Paraguay, "in an increasingly frantic effort to enhance their résumés". The lesson for Britain, from both Marris and Cassen, is that change at the top matters. But change at the bottom matters most of all. An underclass is often perceived as a threat - which it sometimes is. But mostly it is a scandal and a waste. Say not, Arthur Hugh Clough went on, that "as things have been, things remain". All social victories are won by stubborn small steps. For equality of opportunity, so will it have to be again.

Paul Barker is senior research fellow of The Young Foundation

Further reading: Tackling Low Educational Achievement by Robert Cassen and Geeta Kingdon is published by the Joseph Rowntree Foundation