But Swan survived, and on Wednesday, 81 years to the day since Captain Oates took his sacrificial stroll in Antarctica, he was telling the tale at the RGS to an invited audience of more than 150 captains of industry.
'We are glad to be associated with a pioneer like Robert,' said the chairman of KPMG Peat Marwick McLintock, acting as the master of ceremonies, 'because we like to think of ourselves as pioneers of accountancy.' The business folk were there to enjoy the kind of slide show you don't get after your neighbours return from a fortnight in the Algarve.
'Let me get one thing clear,' the 35-year-old Swan began. 'I am not a masochist. I derive no pleasure from waking up in the morning with ice in my underpants.'
Robert Swan has told his tale more than a thousand times. It is a story of courage, determination, and grotesquely frost-bitten digits; of cavalier motivation ('the South Pole was done because it went down well with girls at parties'); of self-deprecation ('the first thing I did when I got to the North Pole was thank God for putting only two poles on the Earth: now I need never do this again'); and public-school self-belief (Swan went to Sedburgh in the Cumbrian hills, where they wear shorts in the sixth form: you don't come more public school than that).
It is also a tale which, since he returned from the North Pole in 1989 with debts of more than pounds 600,000, has become his living. Charging up to pounds 3,500 a time, he has travelled the world with his talk, determined, like a sort of ice-walking Jeffrey Archer, that the debt will be repaid. He sells himself to corporations to gee up their staff. 'Here's how I walked to two poles,' is the gist of his patter. 'If I can do that, you can sell a few more baked beans.' He has become so good at it that Time magazine named him as one of the world's top 10 motivational speakers. Four years on, his overdraft stands at less than pounds 10,000.
'Getting that debt down was far harder than walking to the poles,' Swan explained before the talk, sitting in his office by the Thames. 'That walk was the easiest thing I've ever done. Not physically, obviously. But philosophically. You either got there or you died. But God, what I do now? I came back from the North Pole less tired than this.'
Swan was, in one way, lucky to have incurred the debt. According to Shane Windsor, who runs the RGS's Expedition Advisory Centre, it gave him something to focus his energies on when he returned. Many explorers have found only despair. 'Coming back from a walk across the Antarctic is like ending a passionate love affair,' Ms Windsor said. 'Suddenly, instead of all that emotional charge, of living right at the edge of experience, there is nothing.'
And Swan had quite an experience. After landing in Antarctica, he and his team stayed for seven months in the wooden hut built by Captain Scott. They sat on the dining chairs Scott had thoughtfully taken with him, ate his jam and biscuits, preserved perfectly three-quarters of a century on by the biggest freezer on the planet. Outside the hut they saw the bodies of Scott's dogs, barely decomposed; and they passed the place where Scott must have died, where his body lies in the ice, unaged.
Such experiences are why so many polar explorers cannot keep away. In the past, the first question people asked Robert Swan was, 'How do you go to the loo at minus 60?' (answer, 'As quickly as you bloody can'). Now they want to know his opinion of Sir Ranulph Fiennes; is he barking to keep going back, or what?
Swan is not willing to be drawn into a comparison with Fiennes, whose bravery, he says, is beyond question and who has raised millions for multiple sclerosis. All he will say is he is worried that 'we are going to enter a time when exploration is a bit like the Olympics - competitive stuff about who can go longer, faster, without this, with that. It's what you do with your experience afterwards that is important'.
Now that his debt has been all but paid off, Swan has turned to a new challenge. He has something else to sell: he is drawing up a plan to save the world. 'There is no doubt that Robert had a spiritual experience down there,' said Ms Windsor. 'He returned with the kind of missionary zeal you only normally see in born-again Christians.'
Swan casts himself as 'the Indiana Jones of the environmental movement'. He reckons his conversion came half-way to the South Pole, when he and his two companions smelt burning flesh. It was their own. Nowhere in Scott's journals was there any mention of problems with sunburn, so Swan could not understand why his face was peeling off. But this was 1984, and the hole in the ozone layer had not been discovered. Swan still has scar tissue all over his cheeks to prove that the green scare stories have a point. In 1990 he was made UN Ambassador for the Environment, and in that capacity went to the Earth Summit in Rio. There he was enraged by the gap between environmentalists and businessmen. Hence his lecture to the cream of the CBI.
'I'm 35,' he explained before the talk. 'I don't want to be stuck with the tag as the bloke who walked to both poles all my life. I want to do some serious exploration about how we can adapt to survive. Out there on the ice, it seems very simple: you get your act together or you die. I've been trying to transfer that simplicity to a complex life here. I said that to Jeremy Paxman on Newsnight and sounded like a drunk Zen Buddhist.'
Swan has leapt into his new role with a ferocious gusto. He leads a peripatetic life, too busy to put down roots, sleeping on a futon on his office floor. His latest scheme is for a huge television series, 'a green Troubleshooter' he calls it, in which he will go around the world seeing how industrialists are shaping up to the environmental challenge. And he was asking Britain's chairmen to let him know what they were up to.
'I reckon the new heroes of exploration are not the guys who climb Everest backwards in their underpants,' he said. 'It's the company in Barnsley that pioneers a way of making environmentally sound lavatory paper. We have to make these people feel good, not guilty.'
His invitation to Britain's industrialists to take part in his project - the conclusion to his wonderfully motivating talk - seemed to go down well. 'I'm not sure if I agree with the detail,' someone from BP said, 'but you have to admire his energy.'
The first place Swan plans to visit for his programme is, almost inevitably, the North Pole. 'When I went there, I walked the last 500 miles with a slipped disc. Every time I give a lecture and repeat the story, I get a sore back. When I go near cold, I start to shake, I start to sweat, I go into palpitations. My body says, 'Why the hell are you doing this to me again?'. The pole absolutely terrifies me.'
So why on earth is he going back?
'I want to exorcise the horror,' he said. 'I'd like to be able to look around, be a tourist. Anyway, it may be my last chance before it melts.'