You can't run a fair society from the Gents'

David Cameron has nothing against women, I imagine, in the right place. He's married to one, and he's got a couple of daughters, although he did once leave one of them behind in a pub. I don't suppose he gives much thought to the other women in his life – or rather their absence. He is leader of a political party which has woefully few women MPs, he has a poor record of promoting women to Cabinet and he's notorious for patronising Labour women during Prime Minister's Questions. Remember when he told a Shadow Cabinet minister, Angela Eagle, to "calm down, dear"?

Now he's been criticised by the first female head of the Home Office, the kind of person who very rarely speaks out, for excluding women from top government posts. Dame Helen Ghosh, who left her job as permanent secretary last month to run the National Trust, told students at a Cambridge college that Westminster is run by powerful networks of men which are hard for women to break into. She pointed out that there was a "magical moment" six years ago when half the heads of government departments were women, but now there are only three female permanent secretaries.

Her remarks touched a raw nerve in Downing Street, which responded that Cameron's press secretary and the official who organises his public appearances are both women. But the wider question is whether the Prime Minister's background is having an impact on the appointment and promotion of women in public life. Although he makes an effort to look modern, Cameron comes from a class whose habits are quite peculiar, compared with the rest of us. It isn't just that he's posh, and married to a baronet's daughter to boot. He went to a single-sex boarding school, where learning to network is second nature, and kept up the tradition at Oxford by joining the all-male Bullingdon Club. Cameron doesn't strike me as reflective and it may not even have occurred to him that these masculine environments are neither normal nor desirable. But they undeniably exclude women: "Women don't network. It is actually quite difficult for a woman to get in as part of an Old Etonian clique," Ghosh is reported to have said.

It isn't just politics where it's hard for women to make informal contacts. In one of my first jobs, I worked in an office next to the men's lavatory and quickly realised how many decisions were made by blokes chatting to each other in a space closed to women. The problem with someone like Cameron, I suspect, is that wealth and class have combined to make male elites familiar and comfortable. It's actually weird that there were a dozen Old Etonians in the government last year, if we include government spokesmen in the House of Lords, but I bet it didn't seem so to the Prime Minister. When the country is run by an upper-middle-class white man, diversity and equality go out of the window.

www.politicalblonde.com; twitter.com/@polblonde