We already know David Cameron has a problem with women. But the stories from the sacking of Caroline Spelman suggests he has a particular problem with older women too. And it matters. Not because of one environment secretary or a few senior politicians. But because women across the country are being heavily hit by this Government, and older women are getting particularly squeezed. And we cannot afford a Prime Minister who doesn't get it or doesn't care.

The reshuffle showed David Cameron's blind spot. Despite his promise to increase the number of women in government, the proportion remains a lowly one in six – half the number of the previous Labour government. Cabinet women were far more likely than their male colleagues to be sacked. Departing junior men were rewarded with knighthoods and honours; departing senior women were ignored.

In Caroline Spelman's case, the Prime Minister made it even worse. No 10 has refused to deny that he told the 54-year-old environment secretary she was too old. Yet he replaced her with a 56-year-old man. (The 72-year-old Ken Clarke stays in government too.)

Of course Caroline Spelman is not the first woman to find that age is treated differently for women and men. The pay gap between older women and men is much higher than for younger people. Women are more likely to keep working past the retirement age. However, while men who keep working are most likely to be in senior jobs, women who keep working are most likely to be low paid.

But, far from tackling discrimination, this Government is making it worse. Women are already paying twice as much as men in George Osborne's tax, benefit, pay and pension changes even though they earn less. Pension age changes target women in their fifties, with some women losing a massive £11,000. Older women are seeing the steepest increases in unemployment – up by almost 40 per cent for women over-50 since the election, compared with 5 per cent for the population as a whole.

Women in their fifties and sixties are often holding families together and picking up the pieces as public services are cut. Looking after grandchildren because childcare costs are up and tax credits are down, or caring for elderly relatives as social care services get shredded, they are pulled in both directions. We know about the squeezed middle, but this is the stretched middle generation.

Older women will be a central theme in Labour's Women's Conference this year, focusing on the issues women face in the workplace, families and public life.

The women of the stretched middle are the generation that fought for equal pay and broke barriers. They see how the Prime Minister treats women in government and Parliament. But more importantly, they see how his Government is treating them, and they won't stay silent. David Cameron would be wise to start listening.

Yvette Cooper is the shadow Home Secretary and shadow Minister for Women