Joan Smith: The extremely cross of Saint George

I have lots of enthusiasms but I don't wave them in strangers' faces

Whenever you visit an ancient monument, you're bound to see messages from people who've scrawled or scratched their names on the walls, just to let you know they've been there. You don't have to be a shrink to recognise it as a pathetic gesture, born out of a chronic feeling of insignificance. This is what I suspect lies behind an epidemic of men (it is usually men) driving around with the cross of St George fluttering from their car windows.

What are they trying to tell me? If it's the fact that they're English, so what? If it's that they're England fans, I couldn't care less. I have lots of enthusiasms, but I am able to get through life without waving them in the face of total strangers. The Olympics and the World Cup could take place on another planet, as far as I'm concerned, and I'm utterly bewildered by the debate on whether Scottish fans should support England. I'm English, and I hope England are knocked out at the first opportunity.

This isn't just a reaction to the irritating ubiquity of football trivia. What I love about my country is mostly intangible, its literature and its long tradition of radical politics from Mary Wollstonecraft to John Stuart Mill, but I'm hardly going to dangle copies of Mill's treatises from my car aerial. I grew up reading Tacitus and Juvenal as often as Dickens, and I feel as much European as English. The nearest I get to patriotism is a quiet pride in this country's longstanding commitment to human rights and democracy.

I can't stand the kind of yah-boo English nationalism that accompanies any big football tournament, epitomised by years of puerile insults towards the Germans. Far from identifying with the flag of St George, I associate it with racism, rowdiness and the careless misogyny of thousands of England fans who will pay for sex with trafficked foreign women during the World Cup. Even in the best of circumstances, it's hard to see how someone like me could ever identify with a symbol that's so masculine and Christian. And the appropriation of the flag by football fans has done nothing to redeem it in my eyes.

On the contrary, it stands for all the things I dislike about my country: xenophobia, boorishness and periodic outbursts of self-pity, expressed in diatribes against foreign players and linesmen who are deemed to have cheated our boys out of their rightful triumph. "We was robbed!" has, after all, been the tribal chant of England fans for generations. In the highly unlikely event of their team winning the World Cup, I expect they'd feel cheated out of their habitual sense of grievance.

Behind all this lies a worrying phenomenon, which is the permanent crisis of a particular kind of English identity. I see no evidence of any such soul-searching among the aristocracy or the people who actually run the country, but nine years of Labour government has actually exacerbated the sense of insecurity and frustration felt on council estates and among blue-collar workers. True, they now have access to unprecedented quantities of consumer goods, including cars and foreign holidays, but in return they've had to give up any number of certainties, including jobs-for-life, a decent state education for their kids and dependable pensions.

They feel insecure, marginalised, and, as always in such circumstances, they blame outsiders. It's not just the Daily Mail that identifies asylum-seekers and East European immigrants as the source of everything that's wrong with this country; working-class people complain about living somewhere they no longer feel is their own and what they're expressing isn't a straightforward expression of anti-black racism. They're as unhappy about European immigrants as they are about Asians; the changing nature of these anxieties was brought home to me after the recent local elections, when a young black Londoner astounded me by admitting he has a lot of sympathy with the British National Party.

The Labour MP Margaret Hodge got into trouble during the elections when she warned of unprecedented electoral support for the BNP in East London, but she was proved right. At a time of growing economic uncertainty, few politicians are making the case that the freedom to travel and work throughout the EU benefits us all. Instead, suspicion of foreigners is expressed more openly than at any time since my childhood, when Enoch Powell stoked working-class hostility to black and Asian immigrants. The Cup may have provided an excuse, but it's that sense of insecurity and xenophobia that's leading so many young men to wrap themselves in the English flag.