Robin Scott-Elliot: A spade is a spade – and ideal for gardening leave

The Way I See It: Richards will serve out his time as a lame-duck chairman, collecting his £291,000 salary en route

According to Sir Dave Richards, it is his in-built Yorkshire autopilot demanding he call a spade a bloody shovel that left him in hot water last week. He's sorry but what can you do about your roots?

The chairman of the Premier League – who was acting in a private capacity as Sir Dave, man of the football world, and not as chairman of the Premier League, as the Premier League hurried to inform the world – did not endear himself to listeners at the second International Sport Security Conference in Doha as he took out the shovel and dug himself a bloody great big hole with his views on the origins of the game and how it was "stolen" from England by Fifa and Uefa. And then there was his take on local culture in Qatar and the unforgettable suggestion that the desert nation was "starting to bury your head in the sand".

It is not easy to make Fifa appear wronged but Richards managed it. There are unconfirmed reports of howls of laughter emerging from Fifa House in Zurich last Wednesday evening as the YouTube clip of Richards's watery detour gathered hit after hit. Likewise at Wembley the footage might have provided a welcome distraction from the Football Association's own difficulties; Richards has a place on the FA board but would struggle to top a popularity poll around the national stadium. Schadenfreude is German but we have stolen the word for ourselves. It's not something to be proud of – the feeling, not the stealing – but sometimes it's hard to resist.

And that appears to be the end of the matter for Richards. He has 15 months left until his 70th birthday, whereupon he will relinquish the Premier League chair and retire to the Broad Acres, although presumably not to the blue-and-white half of Sheffield after his tenure as Wednesday chairman did so much to hasten the club's decline. He will not resign over what happened last week and nobody has asked him to consider his position. The Premier League and the FA, of which he is vice-chairman, both "distanced" themselves from Richards' remarks and there seemingly the matter ends.

Richards will serve out his time as a lame-duck chairman, collecting his £291,000 annual salary en route. The Premier League holds its AGM in June, when the chairman is in theory up for re-election should the 20 club chairmen want a change. There is no suggestion of any need to dust off the league's polling booth.

If the other David – Bernstein, chairman of the FA – had made such statements he would have been drummed out of office by now. For Richards a "distancing" is deemed sufficient, and the insistence that he was not there representing the Premier League, even though the conference billed him as chairman of the Premier League having no doubt issued the invitation because he is chairman of the Premier League.

"It is the way I am," said Richards as part of his second apology, one that was almost as nonsensical as his original ramble at the conference. Richards has been chairman of the Premier League for 13 years. A recent report into governance in sport by a committee of the Council of Europe highlighted how long chairmen, presidents and the like remain in situ with sporting bodies. Sepp Blatter is the obvious example – he pre-dates Richards as president of Fifa by a year. The difference between Blatter and Richards, of course, is that Richards is regarded as a figurehead, a footballing version of Prince Phillip or maybe an aged Prince Andrew, whereas Blatter governs. But nevertheless that both men remain untouched in their respective positions insults those who follow the game.

The increase in supporters' groups in recent years is because of the alienation felt by an increasing number of fans. Supporters want their voices heard as, among other issues, they struggle to pay the rising prices demanded to watch the game that their presence in such consistently large numbers has helped make so attractive to the TV companies that fund planet football.

The fate of Richards may mean little to most supporters, and most will care little whether he remains in office until he gets his Premier League carriage clock. Yet the continued presence of a man who left Wednesday in such a heap and has fed happily off the footballing gravy train does not reflect well on the organisation he chairs. It may be a symbolic position, but now it is time for a symbolic act: relieving Richards of his duties. Even better would be for the man himself, someone knighted for services to sport, to decide the time has come to take responsibility for his actions. Then at least it might be said of Richards that nothing in his (sporting) life became him like the leaving of it.