Gordon Brown has a problem with women. Women were his most vocal critics last month when he fought to keep his job and as he desperately reshuffled his government. Now BBC Radio 4 has interviewed a range of former and current female ministers about their views on the Prime Minister. The programme-makers have finished up with a treasure trove of opinions. Virtually every interview provides news and another nail in Brown's coffin.

Gordon's Women, broadcast at the weekend, was perfectly timed because of a growing perception that Brown is a King Lear figure, content only with his raucous, bullying male court. In some productions of the play Lear's court is so wildly macho that the audience begins to have some sympathy for the malevolently scheming Goneril and Regan, Lear's two daughters who turn on their father. Surely there is cause for more sympathy in relation to Gordon's women as they reflect on their tormented lives with a Prime Minister and his bunch of rowdy male allies?

It seems the women feel fairly sorry for themselves. The former Cabinet minister Patricia Hewitt speaks of Brown's inner circle as "almost entirely men and really rather laddish in its culture". Jane Kennedy, who left the Government last month, notes Brown's "darker side". Margaret Jay feels intimidated because he "doesn't make it easy to feel that you've established a personal connection". Kennedy compares him to a mafia boss. The former Europe minister Caroline Flint claims that women in particular are "picked out" for malicious private briefings.

Is this the epitaph for a government that begin its life in 1997 by hailing the number of women MPs that had been elected for the first time? In order to get a clear answer let us take a closer look at the allegations and those making them.

All the disgruntled interviewees are former members of the Government without much chance of a return. Some are leaving the Commons at the next election. It would be just as easy to interview a group of former male ministers drained of political ambition or hope who would be equally forthright. Critical candour tends to arise when MPs no longer seek the patronage of their leader. It has nothing to do with the sex of the critic and much more with the narrowing of a leader's patronage after a long period of one-party rule.

Similar views regarding Harold Wilson's court can be found from the late 1960s onwards when some MPs felt, wrongly, that Wilson's days at the top were numbered. A sense of bullying exclusion was a common theme when Margaret Thatcher's critics, male and female, dared to speak out towards the end of her leadership. Tony Blair too was subjected to the same criticism about his male court, not least from one or two women who had been sacked in reshuffles.

In particular there were criticisms of his appointment of Alan Milburn as chair of the party on the grounds that he was too macho, one of the more irrational onslaughts on someone who had previously left the cabinet to spend more time with his family.

Gordon's Women follow the pattern of candour at the end of a limited ministerial career. Flint was famously loyal to Gordon on the night the polls closed after the local and European elections. She fumed against his machismo only after she had not been appointed to the Cabinet. Hewitt left the Government after Blair departed. Kennedy departed at the last reshuffle.

Their criticisms are conveniently imprecise. It is all vaguely about tone and style. They are bothered by "briefings" against them, although it is not at all clear they have been direct victims of smears. There is also, I note, an imbalance in the "briefings" wars. Apparently they can speak out against Brown and his allies, but when it is the other way around they protest about too much machismo. Recently several advisers and ministers have been privately critical of the Brownite minister Ed Balls on the grounds that he has made hostile briefings to journalists. They brief against Balls about alleged briefings. Apparently that is all right.

Most of this minor traffic goes on in politics all the time and has nothing to do with being male or female. For women to attack Brown on the grounds that he attacks them seems to be having your cake and eating it.

The women also conjure up an image of a Brownite court which is largely mythological. We know all about macho Damian McBride and Charlie Whelan. Brown's misguided dependence on these media handlers caused him far more damage than any of their supposed victims, a number that has always been exaggerated.

As far as there were victims they have tended to be male. Probably Alistair Darling has more cause than most to lick his wounds. He is not a woman. Beyond McBride and Whelan there is not a huge amount of machismo in Ed Miliband and several others who were part of Brown's entourage for more than a decade.

A woman, Sue Nye, has run his chaotic office since 1992. One of the most influential figures to whom Brown turns is Shriti Vadera, a woman who appears on the Radio 4 programme to point out that there is a fair amount of diversity in the Government. Angela Smith, who has been a parliamentary aide to Brown, says she never felt held back because she was a woman. Vadera and Smith are both still active in the Government. The real divide is more between the possessed and dispossessed than between men and women.

Even Brown's closest allies accept that he is hopeless at people management, insensitive to ministerial figures, both men and women. John Major was more thoughtful in his personal dealings and yet in his first Cabinet he omitted to appoint a single woman.

Who was the more macho – the well- mannered Major or the supposedly anti-women Brown? There are many epic flaws with Brown – and with the New Labour project – but women who complain about their treatment need to look closer to home in order to find why they have not flourished at the top of politics.

s.richards@independent.co.uk