Bad enough having Britain's media bring up your £10m pay package every time your employer is mentioned – but the real problems for Stephen Hester start when he gets home.
The Royal Bank of Scotland boss's mum and dad think their son's take-home is, well, just a little excessive.
"If you asked my mother and father about my pay they'd probably tell you it is too high," he said, doing his best to look contrite before a hearing of the Treasury Select Committee yesterday.
"So even some people close to me have that view of bankers. I wasn't asked to do this job. I didn't apply for it. I was asked what it would take for me to do the job... All I've ever said about my pay is that I would be content to be paid the going rate to do the job."
It's that "going rate" that bothers people. Mr Hester hardly comes from the backstreets, but nor could it be said that he was born with a silver spoon in his mouth. A grammar school boy, his father is a semi-retired chemistry professor at the University of York while his mother is a psychotherapist.
There was always a gulf between what was earned by middle-class people like them and the small band of pinstriped plutocrats in the City. But in recent years, thanks to a bonus culture largely imported from the US, that gulf has turned into a chasm. And what really rankles is that many of those bankers are now effectively in the public sector. The same people who pay Hester Snr's wages now look after Hester Jnr.
As the chairman of the committee John McFall said: "The sheer scale of your package seems out of synch with what is happening out there. Surely you now rank as the UK's highest paid public servant?"
A small but growing band of top executives appear to have taken note of Mr McFall's point and the fact that this chasm is causing deep resentment. HSBC's chairman, the deeply religious Stephen Green, has even written books about it (one wonders how he finds the time). Wall Street banks have been desperately trying to stave off another wave of fury as they prepare for what could be a record round of bonus payments beginning with JP Morgan on Friday.
Goldman Sachs – whose chief executive Lloyd Blankfein sparked outrage by describing himself as "just a banker doing God's work" in an interview – is considering forcing its richly rewarded staff to make charitable donations.
But donations are a long way from what Richard Harvey, the former chief executive of insurer Aviva, did when he gave it all up to spend a year working for charity in Africa (he's now back in the boardroom, although still heavily involved in the charitable sector).
Mr Harvey aside, the question that has to be asked is whether all this belated humility is sincerely felt or just PR, designed to deflect the growing criticism of the behaviour of the banks.
And, perhaps, ease the pressure for more punitive taxes along the lines of the 50 per cent rate levied on bonus pools by Alistair Darling.
Few British executives have, after all, followed the example of Warren Buffett or Bill Gates, who have pledged to give much of their fortunes away.
Mr Hester went on to say that there have "been instances of pay which have not been justified and I completely understand the public interest in that and the political interest in that". But he would not comment on the reports that RBS plans to pay out £1.5bn in bonuses this year all the same.
And while his package is not worth much at the moment (the bank's shares have to show some signs of life for him to hit the jackpot), with a 350-acre estate in Oxfordshire, a chalet in the Swiss ski resort Verbier and a house in Holland Park to maintain, he's not offering to give it back.
Maybe it would be easier for Mr Hester if his family was more like the Dimon family on Wall Street. It's a fair bet that if Theodore Dimon thinks his son Jamie is paid too much as chief executive of JP Morgan, he doesn't say so. That's because Dimon Jnr is his boss, having poached his 78-year-old father from rivals Merrill Lynch. If you're going to be a fat cat, far better to keep it in the family.