Dunedin will host England's next two rugby matches – but it was designed by the Scots, says Adrian Mourby

Should you ever be asked the location of the world's southernmost statue of Robbie Burns, the answer is: in the middle of an octagonal piazza in the city of Dunedin, New Zealand. I know this because the only substantial human settlement south of Dunedin is on Antarctica and the treaty of 1961 forbids such structures.

After last weekend's victory over Argentina, followers of English rugby will focus their attention on Dunedin twice more, as the team attempts to negotiate its way past Georgia (tomorrow) and Romania next Saturday. However, any fans who've made the 12,000-mile journey to the city to witness the games will soon learn that their traditional rivals got here much sooner. The octagon on which the Bard of Ayrshire sits was designed in Scotland in the 1840s, at a time when Dunedin was a home from home for Scottish emigrants to New Zealand.

A plan for the city was laid out on paper in Scotland and given the old Gaelic name for Edinburgh, Dun Eideann. There were high hopes of this port settlement, tucked away for safety inside the sea-washed crater of an extinct volcano. At first life was hard, but then came the Gold Rush of the 1860s. Rich seams in the Otago Peninsula made Dunedin's fortune.

Even today, Dunedin impresses. The railway station by Sir George Troup was built without a budget. The Edinburgh architect was just told to build the best, so he shipped in granite columns, Royal Doulton tiles and Italian mosaic. It's not surprising the local tourist board claims this is New Zealand's most photographed building. The platform, with its delicate glass and iron canopy, is so long that you are already well on your way to the South Pole by the time you've walked it. Sadly no trains arrive these days.

Much of Dunedin was constructed in a similarly baronial style. The main university building resembles Sandringham. Larnach Castle – home of the tragic banker William Larnach, who killed himself when he faced ruin – is as rugged as the promontory on which it stands. Olveston House, in Dunedin's sumptuous suburbs, is a time capsule of Victorian New Zealand. This immaculate mansion was built by a Jewish merchant who moved to Dunedin from Bristol and increased his fortune importing pianos for the city's ever-growing bourgeoisie. The daughter of the house kept it unchanged after her parents' death, and today it is much-visited by tourists.

On nearby Hawthorn Avenue – but closed to the public – stands the last house that Captain Scott stayed in before departing New Zealand on the expedition that cost him his life. Much of Dunedin is a memorial to the days of Empire. The main thoroughfare of Dunedin is Princes Street. Before strolling down this I peek into the cathedral, which looms over the octagon and is tellingly incomplete. The mock-Gothic nave was begun before the First World War but by 1918 there was no money to complete the original design, which was on the lines of a Hereford or Worcester. The cathedral was eventually finished – without its steeple or transept – in 1971. At the end of the Victorian nave visitors find themselves among angular concrete. A large Perspex cross hangs over the altar.

Princes Street managed the transition to the 20th century more successfully, mainly because its monumental banks remained in situ, gathering dust as the city gave up its aspirations to dominate New Zealand's economy. The Savoy, once the finest place to dine in Dunedin, is now being slowly restored. The Old Bank of New Zealand, a splendid neo-Renaissance structure from 1883 with not a single stone undecorated, is to be redeveloped as an apartment and restaurant complex. Donaldson's Department Store (established 1862) still dominates one city block, with the Salvation Army and Speight's Brewery visible behind. Poor Hotel Central was cut in half along time ago for an aborted new development. Its once impressive facade now reads "Hotel-".

Wain's Hotel, now the Mercure, has fared better. Its brightly painted, bay-windowed façade looks as if it belongs in Eastbourne while the old granite Post Office appears heavy enough to sink the whole city. There have been talks about turning this building into a hotel, but Dunedin's problem is that people come en route to the Antarctic or the Otago Peninsula. The city is a gateway to superb outdoor holidays. That kind of clientele are not looking for bellhops and five-star luxury.

What saved Dunedin after the gold ran out was cheaper Art Deco constructions in the 1920s, neatly filling in the gaps between banks. The long white bus station built in International Art Deco extends below the Post Office on land reclaimed from the sea. It looks like a film set.

After Art Deco, this strange, Scottish city couldn't afford to rebuild further, which is why it's still such an architectural delight today. The one major exception is the Otago stadium, which lies close to Dunedin's Water of Leith. It was opened by New Zealand's Prime Minister John Key just last month. It's a long way from Murrayfield Stadium.

Travel essentials: Dunedin

Getting there

* Air New Zealand (0800 028 4149; airnewzealand.co.uk) has the only direct flights from the UK, with daily departures from Heathrow to Auckland via Los Angeles and five flights a week via Hong Kong; you can include these cities in a round-the-world itinerary. Other airlines offering one-stop flights to Auckland include Cathay Pacific (020-8834 8888; cathaypacific.com), Malaysia Airlines (0870 607 9090; malaysiaairlines.com) and Thai (0844 561 0911; thaiairways.co.uk). Emirates (0844 800 2777; emirates.com) serves Auckland and Christchurch from its hub in Dubai, with connections from Heathrow, Gatwick, Manchester, Birmingham, Glasgow and Newcastle. Singapore Airlines (0844 800 2380; singaporeair.com) has one-stop links from Heathrow to both cities.

Staying there

* The writer stayed at St Clair Beach Resort, Dunedin (00 64 3 456 0555; stclairbeachresort.com), where doubles start at NZ$195 (£102), room only.

* Mercure Dunedin (Wain's Hotel) 310 Princes Street (00 64 3 477 1145; accorhotels.com). Doubles start at NZ$139 (£72), room only.

More information

* Tourism New Zealand: 020-7930 1662; newzealand.com