There was something predictable and depressing about the reaction to David Cameron's "Calm down, dear" rebuke to Angela Eagle in the Commons last week. Sexist! Patronising! Insulting! The piping from some, mainly women, Labour MPs quickly turned into a full-blown symphony about sexism in Westminster.

I cannot agree that either the remark itself or the Prime Minister were sexist. But on the wider issue, after working 10 years as a journalist in the Commons, I feel the image of the gentleman's club, wheeled out again last week, is no longer relevant.

Mr Cameron's put-down was patronising – but put-downs at Prime Minister's Questions are supposed to be. He once deployed the same phrase against David Miliband. So, if he had pulled his punch against Angela Eagle, wouldn't he be guilty of condescension, as if a woman cannot take the same jibe as a man? And wasn't Ed Balls equally patronising, in demanding an apology from Mr Cameron, as though he were in a pub asking another bloke, "Are you looking at my wife?"

Was this Mr Cameron revealing his true colours, as an Old Etonian who believes women should return to the kitchen? Despite the pictures of Samantha Cameron in a pinny, making cakes last week for the Downing Street royal wedding party, the Prime Minister's wife is one of three strong, independent-minded women that he uses for advice and support. The other two are his press secretary Gabby Bertin, one of his most trusted confidantes, and Catherine Fall, his deputy chief of staff, to whom he is also incredibly close. He believes, say friends, you cannot have a "360-degree view of life without having 50 per cent of the population represented in your inner circle".

On a personal level, Mr Cameron talks to women and men in the same way. The same cannot be said of his two predecessors. Off-camera, Tony Blair had a forced, unnaturally flirtatious way about him, while Gordon Brown seemed to find it difficult to talk to women at all. Mr Cameron is neither flirtatious nor uncomfortable around women. Smooth and sometimes haughty, yes, but this manner is applied to men and women alike.

As for sexism in Westminster, I accept it remains in some corners of the Palace, but it is fading. Over the past decade, I can recall only two or three examples of off-colour remarks, which were more amusing than horrifying. A Tory MP once, rather sweetly, told me my legs made him "proud to be British", while a Labour Cabinet minister fell over a chair in his office as he rushed over, blushing, to tell me it was "so lovely to see" me. But, with each election bringing a younger generation of MPs to mill around the Portcullis House atrium, the atmosphere in the Commons in 2011 is more like the offices of Google than the stuffy rooms of the RAC Club. Female and male MPs who have families are finding the hours friendlier.

To complain that the Commons is too shouty, too male, is to suggest that women cannot cut it in this environment. Yet Bessie Braddock could hold her own against Winston Churchill 60 years ago. And Harriet Harman – who used to appeal for more consensual debates in the Chamber – was best deputising for Gordon Brown at PMQ when she deployed humorous and withering tickings-off against her opponent William Hague.

To jump on every remark as sexism devalues the equality argument and risks turning Westminster women into victims. Female MPs should fight the real battles – on equal pay and the effect that government cuts will have on the work-life balance – and brush off the silliness of a rather poor Michael Winner impression.