My purest Joseph Conrad moment in Malaysia came after a dawn flight from the island of Penang down the coast of the peninsula to Kuala Lumpur. Half-awake, I stumbled into the KL arrivals hall to find it a sea of white robes. The crowd trailed cardboard monoliths of baggage tied in string: they were hajjis, just returned from Mecca.

In George Town, the port of Penang, I had come across Acheh Street: a century ago, the hub of the pilgrim trade. There, inns and agencies – often run by Arab traders from Sumatra – catered for Muslim Malays on their way to Jeddah, and then Mecca, via European- or Chinese-owned passenger boats. On the corner of Acheh and Armenian Streets, the handsomely revamped Syed Alatas mansion now acts as a museum of Muslim Penang. It belonged to an Arab-Sumatran magnate whose first wife came from Malay royalty; his second, from the Straits Chinese community. In this port, one man's faith became another man's business.

An ordeal suffered by those pilgrims' ancestors, 130 years ago, lay behind Joseph Conrad's great Lord Jim. In that novel (published in 1900), Conrad gives the "water-clerk"-turned-sailor Jim much of his own restless itinerary as a merchant mariner as he knocks around the eastern seas from port to port – Bombay, Rangoon, Calcutta, Penang. Then, as first mate of the pilgrim-ship Patna ("owned by a Chinaman, chartered by an Arab"), Jim gets a taste of authority. The ship takes on a complement of hajjis: "Eight hundred men and women with faith and hopes, with affections and memories".

Before journey's end, in a panic, the officers and crew will abandon their passengers, lying that the ship has sunk. It has not; the deception is unmasked. Branded with shame, Jim flees to seek redemption as the benign ruler of a jungle outpost. Two real histories fuelled the novel: the fate of the SS Jeddah, which sailed from Penang in 1880, and the life of James Brooke, the "White Rajah" of Sarawak in Borneo.

Penang's romance coexists with no-nonsense reality. Conrad buffs will know that more of his south-east Asian plots unfold in Bangkok or Singapore. But you will never track down the mood of the entrepots he often merely calls "an Eastern port" in those concrete metropolises now. Nor will you, for all the powder-sand beaches and top-notch accommodation, amid the shopping malls, high-rise condos and luxury hotels of the seaside area of Gurney Drive and Batu Ferringhi. But as for George Town's treasure-crammed historic core, protected since 2008 – with its warren-like lanes of artisan-filled "shophouses", tucked-away temples and gleaming Regency-era civic monuments – by its listing as a UNESCO World Heritage site: this is Conrad's "Eastern port", still intensely present and correct.

Don't, in down-to-earth Penang, expect celestial bliss of the sort Conrad's sea-dog Marlow feels on sniffing the aromatic odours of the East in Youth, "a whispered promise of mysterious delight". In George Town, founded by by the trader Francis Light for the East India Company in 1786, rapture takes on a more solid – often edible – form. Ah, the flavours of Penang! Here, even mystic Marlow might meet his match. The cuisine, like the cultures, delivers not so much fusion as dialogue – give-and-take, free exchange. No style dominates. Sample the uniquely spiced "Peranakan" dishes from the mixed Chinese-Malay community; traditional fare from Fujian, land of their forefathers for most Hokkien-speaking Penang Chinese; Indian Muslim "Mamak" curries; local classics from char kway teow to nasi kandar, served fast in street cafés, in grand colonial dining-rooms (try Sarkies Corner at the venerable E&O Hotel), in high-decibel food courts among lunching bankers (try Weld Court on Lebuh Pantai) and at the ubiquitous hawker stalls.

Even religious transcendence takes on the form of workaday nourishment. Half way down Jalan Majid Kapitan Keling – the "street of harmony" that runs past Anglican church and Hindu temple to the main mosque – stands the Chinese Goddess of Mercy temple. Beyond the smells and bells, candle-crowded shrines and bulbous red lanterns, shoppers gather for a spiritual pit-stop as functional as a cup of tea. Your eyes sting with smoke from a thousand offerings, but it makes for a cheerful, distinctly this-worldy sanctuary.

Here's another flavour of George Town that ties it to Conrad's eastern world. More of old China survives within a tight space here than in the bulldozer-happy Middle Kingdom. Enjoy it in a batch of gorgeously high-impact buildings: dragon-swathed carnivals of the gilder's, carver's sculptor's and glazier's arts. Visit the Cheong Fatt Tze mansion, the "Blue House" built by a mighty shipowner; and the Khoo Khongsi, the exuberantly grand clan headquarters of the Khoos (along with the Tans, Lims, Cheahs and Yeohs, the "five great clans" of Penang). For a gentler atmosphere, track down the green-walled Peranakan mansion on Lebuh Gereja. It was developed around 1900 by the Kapitan Kina Chung Keng Kwee: leader of the Chinese-Malay community whose eclecticism his home celebrates in an east-west riot of antique-filled rooms around a magical central courtyard.

Then, for a cool neo-classical contrast, visit the Palladian mansion of Suffolk House, elegantly restored in 2007. Here Francis Light lived with his companion Martina Rozells: part-Siamese, part-Portuguese, and one of those unofficial mates who fixed the deal and cleared the path for every British empire-builder in the East. So many kinds of people wrote this island's story. Try to find a taste of all of them.

Boyd Tonkin travelled to Malaysia with long-haul specialist Hayes & Jarvis (0871 664 0246; and stayed one night at Parkroyal Penang, two nights at Shangri-La's Rasa Sayang Resort and two nights at the Hard Rock Hotel on Penang, followed by two nights at Traders Hotel Kuala Lumpur. Departing on 10 May, a seven-night Hayes & Jarvis package based on the same trip costs £1,189 per person (based on two sharing) including Malaysia Airlines flights from Heathrow, transfers and room-only accommodation. A seven-night holiday (five nights at Bayview Beach Resort, Penang and two nights at Hotel Capitol, Kuala Lumpur) costs from £899 pp, departing 10 May with Malaysia Airlines

Malaysia: tea, noodles & orchids

* Malaysia's urban food markets are where its cuisine shines – a mouth-watering mix of Chinese, Indonesian, Indian, Thai and Malay flavours. In Kuala Lumpur, Jalan Alor Street comes alive at dusk, lined with stalls selling everything from Chinese noodles to the Malay staple, ikan bakar (grilled fish);

* Head to Tioman Island, off the east coast of Malaysia, to stay at Japamala Boutique Resort. Set in coastal forest, facing a white-sand beach, this resort has treetop chalets, a very swanky new two-bed villa built into the cliffs, with private pool;

* Escape the humidity in the Cameron Highlands. This hill station, set 1,500 metres above sea level, was mapped by the British in 1885. Mock-Tudor hotels offer cream teas and G&Ts plus trekking excursions through a landscape of tea plantations, strawberry farms, and rich butterfly species.

* A truly modern Malay retreat, the Detai crowns the island of Langkawi off Malaysia's northern coast. Set in the rainforest above a half-moon cove, the hotel comes with private pool suites and villas on stilts, decorated with Malay carvings and antiques;

* Retreat into the rainforest. Maliau Basin is a remote area in the Malaysian region of Sabah (Borneo). Enclosed by cliffs and plunging waterfalls, this natural wonderland is home to 12 types of forest, 80 orchid varieties and 270 bird species. Tours from Malaysia Experience (01249 467164,