The House of Commons Public Accounts Committee has taken yet another potshot at the organisation for not being tough enough on them.
Its latest report voices concern over a growing perception that there is one rule for the rich, and one rule for everyone else when it comes to paying tax.
The fact that those in the former group have their own customer relationship managers only fuels that perception. Throw in that they are paying less tax than a few years ago and rarely seem to face prosecution, and it's not hard to see where that perception comes from.
HMRC has tried to give the impression that it has been taking a tougher line of late. Over Christmas it emerged that it had slapped down a scheme that saw a group of wealthy individuals investing in empty data centres up in the North East with the aim of making a tax profit. The fact that footballers and celebs were involved and were told to pay up, represented a PR win.
But, having investigated HMRC's claims, the PAC, chaired by Labour's Meg Hillier, says they don’t stack up.
Its report says that despite HMRC having set up a special unit to deal with 6,000 or so high net worth individuals in 2009, the amount of income tax they have paid has fallen by £1bn since then. The amount the wider public has coughed up has risen by £23bn over the same time.
The PAC says that a third of all high net worth individuals can find themselves under investigation at any one time, with cases that could be worth up to £1.9bn currently open. And yet, since 2012, HMRC has issued 850 penalties totalling just £9m.
The average penalty works out at £10,500, and as the report points out, that’s hardly going to change the behaviour of multi-millionaires.
For them, it’s little more than a rounding error, and they and their advisors, are getting more devious.
While off the shelf schemes, like the one I mentioned above related to the data centres, have been attacked by HMRC, wealthy individuals, aided and abetted by clever advisors, have simply switched to using tailor made schemes that do much the same thing.
This amounts to their trading in their Primark togs for Saville Row suits, and so doesn’t come cheap. But if it keeps money out of the exchequer, and in their bank accounts, they're not going to worry about some extra fees.
Given the prevalence of such bespoke schemes, its surely time for the advice handed out by HMRC’s customer relationship managers to be recorded, just as it is when you and I contact an HMRC call centre (if we can ever get through).
Football comes in for particularly cutting criticism in the report, as a result of the use by some players and clubs of image rights as a means of cutting tax bills, and it is frankly scandalous that some are still not providing HMRC with data having struck a voluntary agreement with the English Premier League.
Greedy Premiership players and clubs are wont to raise hackles at the best of times.
But the issues surrounding that particular sub sector of the uber wealthy should not detract from the wider issue of whether HMRC is going about its job in the right way, and the need for the organisation to take a long, hard, look at its approach and at how well is working.
It might also be time for Government to take a look at whether HMRC is working. These reports from the PAC are published with a disturbing regularity but HMRC appears to feel that it can ignore them (it snarled in response to this latest one). That's not good enough.
If confidence in the tax system is to be maintained, something has to change, and quickly.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies