It was the start of a lifetime of taking bets on the direction of the world's financial markets that has made him a fortune and given him the freedom to do what he wants with his life.
Although virtually unknown in this country, in the United States, where investment is regarded as an honourable occupation, Mr Rogers is something of a phenomenon.
A self-confessed "loner and misanthrope", he made his name - and his money - as George Soros' partner in the early years of the Quantum Fund, one of the most successful investment partnerships ever.
In what appears to have been a profitable but strained relationship, Mr Rogers did much of the research while Mr Soros (so he says, anyway) made the big strategic decisions.
In 1980, when he was 37, there was some sort of falling out and Rogers cashed in his share of the partnership and "retired".
He now invests as a hobby rather than as a professional money manager.
In his spare time, he teaches a class on investment at a business school in New York, and makes regular appearances on television programmes and in the investment pages of the financial journals.
Some day someone will write a learned academic thesis about the influence of Oxford University on modern investment trends.
Like Sir John Templeton before him, another American who spent time as a postgraduate at Oxford, Mr Rogers can justly claim to be one of the pioneers of today's hottest fad, investment in emerging markets.
Just as Sir John was one of the first to spot the economic potential of Japan when it was still an emerging economy back in the 1960s, so Mr Rogers has been demonstrating for years just how profitable picking the next Japan or Chile can be - if you can find them.
At the moment, as it happens, there is plenty of supply. With communism discredited, and trade and investment barriers coming down all round the world, there is no shortage of wannabe capitalist nations.
From the Andes to the Urals, countries that once spent their time bleating about First World exploitation are now opening their markets and concentrating on getting rich. Even those who are still socialist, says Mr Rogers, now want to be rich socialists.
Five years ago, aged 48, Mr Rogers fulfilled a lifetime's ambition by leaving New York to travel round the world - some 57,000 miles in 20 months - by motorbike.
On the way, flogging through the outer reaches of Siberia, Australasia, Africa and South America, he found there were simply too many investment opportunities to resist.
If he liked a country - and he liked New Zealand, Argentina, Peru, Botswana, and several more - he went down to the local stock exchange and bought a bunch of blue-chip stocks.
In the case of Botswana there were only seven shares altogether, so be bought them all from the only broker the country had.
The results of this journey are chronicled in a splendid but unusual book, Investment Biker, half travelogue, half musings on the state of the world and its current investment opportunities.
Anyone interested in how smart investors tick, or how to read the entrails of a market, will find it full of valuable insights. In Mr Rogers' view, all markets - whether you are talking about stock markets, currencies, gold, or wool - ultimately dance to the same tune.
Like nearly all the most successful investors, Rogers is an unashamed contrarian.
Just as trees can't grow to the sky, goes one of his favourite aphorisms, so markets don't go in the same direction forever.
The investments he likes best are those that are currently most out of favour. What he looks for in countries are economies which are experiencing "secular change", but whose moves towards economic realism have yet to be fully appreciated.
Best of all are those that have recognised the need to attract foreign capital and are just beginning to develop investor-friendly stock markets.
Getting in on the ground floor is the way to get the most value out of these situations.
The one condition that he insists on is that the country must have a convertible currency.
This is essential if investors are to have any confidence in being able to get their money out when the time arises.
The black market rate of a country's currency is generally a good guide to the health of its economy.
In Mr Rogers' case, that means directing his money towards several countries that would not feature even on the list of most specialist emerging market funds.
Not many funds are yet ready for Ghana, Uruguay, Botswana and Peru.
His tip for the hottest markets of the next decade are (of all unlikely places) Iran and Venezuela.
By contrast, he is bearish about Mexico - a "sham" - and doubtful about the prospects for Eastern Europe and the Hong Kong market. Zaire he would avoid like the plague.
Not many ordinary investors, it is safe to say, will actually feel the urge to follow Mr Rogers' lead and put their money in these out-of-the- way places.
But you should certainly note the thinking that underlies his investment approach. His view is that while there are huge opportunities in investing aboard, the great bull market in American and UK shares of the last 20 years is now in the process of drawing to an end. He fears that the twin deficits in the US are symptoms of a deep-seated economic and social malaise in his home country.
When I spoke to him in London this week, his view was that we are moving into a period when commodities and natural resources are about to experience a renaissance.
They have been in a bear market for at least a decade, and even longer in some cases.
The places to invest in future are those that will benefit from this next great secular change.
It is no accident that many of the countries he likes from an investment perspective have a wealth of natural resources.
But the best investment advice anyone can give to their children, says Mr Rogers, is to tell them to learn Chinese or Spanish.
Just as the last century belonged to the British, and this one to the Americans, so the next century will inevitably belong to the Chinese.
The way to invest in this change is probably not to put your money directly into China (where the political and economic environment remains uncertain), but into countries run by the "offshore Chinese", where the rules of capitalist life are better understood.
Investment Biker by James Rogers. John Willey. pounds 12.99.