Two balding men who look remarkably alike went head-to-head in the Commons today, trading statistics and insults with a pleasing symmetry.

Iain Duncan Smith yet again reminded Liam Byrne of the post-election letter he wrote (regrets, Byrne must have a few) as outgoing Treasury chief secretary to his successor saying "I'm afraid to tell you there's no money." And he claimed that Labour "spent taxpayers' money like drunks on a Friday night, with no care or concern for how effective it was." (This seems unfair on Friday night drunks, who usually have a shrewd idea what they are paying for.)

Byrne reminded Duncan Smith of when he used to visit Glasgow's most impoverished estates and affirmed that Conservatives would be judged on their treatment of the poor. Now he insisted, the "sage of Easterhouse" had been suborned by George Osborne to conduct a "hit and run" on low-income families.

But, despite these routine exchanges, time may finally be running out for George Osborne's blinds, behind which, he likes to indicate, dole claimants lurk fast asleep, enraging their hard-working neighbours.

Before, there was an unholy Labour/Tory alliance of rhetoric which marginalised the unemployed. The Tories argued that their Bill to uprate benefits by a well-below inflation one per cent will penalise the workshy. Labour, armed with evidence that several thousand poor working families voting in Tory constituencies will also be hit, called the measure a "strivers" tax, hardly daring also to attack cuts in jobless benefits.

But whether because Nick Clegg this week denounced the division between skivers and strivers – while backing the Bill – or even some Tory backbenchers protested at it, or Labour's Harriet Harman earlier said "lying in bed with the curtains drawn" was "no way to speak about unemployed people" the image seems to be going out of fashion. Duncan Smith avoided it.

The most telling speeches came from backbenchers like Labour's Karen Buck who pointed out that many of the poor "churn" between benefits and work whenever they can find a job. Or the former Lib Dem minister Sarah Teather who last night voted against the Bill. And indeed from David Miliband, who excoriated this "rancid Bill", pointing out that while Government said no one earning more than £26,000 should receive benefits, they gave tax relief of pensions up to £40,000.

The "enemy within" was "unemployment not the unemployed" he said. Goodness knows whether he will return to the front bench. But it was a speech that made you think it would be no bad thing if he did.