Once a military base, Governors Island now serves as a summer retreat for daytrippers from Manhattan. Kunal Dutta takes a trip across the Hudson

It is the sort of existential denouement befitting of a Woody Allen film. The last time Katherine and I were in New York we were perched on a rowing boat in Central Park two summers after 9/11. Young, naïve and barely in our 20s, I was dreaming about our children's names and who we might invite to our wedding. She – I would later discover – was plotting her escape to a better life.

At around the same time, the Bush administration had just passed through a protracted deal concerning the future of a scraggy, redundant military base marooned off the southern tip of Manhattan. For years, authorities had agonised over what to do with Governors Island, even reviewing multimillion-dollar bids from private property developers to transform it into a decadent gambling hub akin to Atlantic City. It was only after a campaign by local pressure groups – and thanks to a late push by Bill Clinton towards the end of his term – that the island was sold to the state of New York for a token sum of $1.

The only condition was that it was to be developed into an outdoor cultural destination. So when we chose Governors Island as the spot to reunite as friends eight years later, both of us were too old and jaded to be sold by the flamboyant optimism of the attendant handing us bicycles. "The couples that were dating in Central Park a decade ago," he said, with that brazen North American tone that ends each clause as if asking a question. "They now come here with their young families."

Yet the 140-acre island – which has green fields, expressionist installations by the American artist Mark di Suvero and nearly three miles of cycle track – is a widely different offering to Central Park.

Firstly, it remains largely unknown (ask most New Yorkers about the latest outdoor attraction and, chances are, they'll mention the High Line, a new elongated strip of green that runs across the city's West Side).

Secondly, you need to cross just under half a mile of the Hudson River to get there. The free ferry that departs from the Battery Maritime Building each weekend sinks a foot or two with the sheer weight of numbers. It is packed with upwardly mobile households from Manhattan and Jersey City. Dads stay downstairs encumbered with babies and bicycles. BlackBerry-clutching "hockey mums" keep one eye on their emails and the other on their toddlers, and many passengers head to the back of the boat as it leaves the lower-Manhattan skyline behind.

The elderly woman next to us was revisiting the island where her Sicilian ex-boyfriend was held as a European prisoner of war during the early 1940s. What became of him? I asked.

"I really don't know," she said. "But the joke still remains: what's an Italian soldier's best manoeuvre?" She flung her two hands into a surrender position and chuckled uncontrollably.

In a journey time of just 10 minutes we were transported back in time. Governors Island is best known as a former strategic base for British and American forces. During the American Civil War it was used for captured Confederate soldiers and this year visitors can explore the circular red sandstone fortress to stare into the dank cells where prisoners were housed.

The military symbolism remains. During the First and Second World War the island served as a critical supply base and, later, it was where the former president Ronald Reagan hosted a summit with Mikhail Gorbachev near the end of the Cold War.

At Soissons Dock, free bikes are available in numerous varieties, ranging from sensible to comedic (couples can ride in tandem while families should test-drive the quadcycles).

Cycling around the island is like wandering through a set of Scooby Doo. The abandoned military dormitories that were halls of residence for Confederate soldiers remain behind wrought-iron rails. Other buildings on the island are more akin to that of a 1980s' council estate in Tower Hamlets.

During our ride we stopped off at Noland Park, bedecked with trestle tables where children make art and crafts. Geese wandered through the grounds, while parents picnicked beneath the shady oak trees. Nearby, an unconventional golf course at Liggett Terrace is testament to an island that fuses its past with progressive (and often esoteric) art. There are very few places in New York where you can stand on ground and see the Statue of Liberty, Brooklyn Bridge, downtown Manhattan, Jersey City and Ellis Island from one spot, yet from the Great Promenade this view is available without extortionate sightseeing fees.

But it is the fusion of art and history that gives the island a different vibe to anywhere else in New York. On Colonel's Row the island is decadent, leafier and more salubrious, featuring detached Georgian-style mansions situated in gardens with old-fashioned iron lampposts. The ability to wander between the eclectic areas, ranging from decadent to dilapidated, gives the place an esoteric charm.

How long it will stay this way, though, remains to be seen. The island is in the final throes of a 10-year, $330m (£205m) investment, with a renovation programme that could see some of the empty buildings bulldozed and ambitious plans to transform the island into a recreational playground.

Proposals include a hotel, conference centre and an extension to New York University – and all are under review as part of a wider consultation, which means the island could have a very different feel when construction begins late next year.

As we returned to Manhattan, the fresh bluster of the Hudson River transformed into the still and sticky heat of New York City.

Weary daytrippers lumbered off the boat and Katherine and I parted ways. Ten years after 9/11, New York is rapidly moving into a phase of renewed self-confidence and the next time we meet the island could be another airbrushed, well-heeled part of the city's picture-postcard offering. But for now, this rugged work in progress remains caught in time. And it is available – in this transitory stage at least – for just one special summer.

Travel essentials

Getting there

New York JFK is served from Heathrow by British Airways (0844 493 0787; www.ba.com), Virgin Atlantic (0844 874 7747; www.virgin-atlantic.com), Air India (020-8560 9996; www.airindia.com), American Airlines (020-7365 0777; www.americanairlines.co.uk) and Delta (0845 600 0950; www.delta.com). BA also flies from London City. From Manchester, JFK is served by Delta and American Airlines.

Newark is served by BA, Virgin Atlantic, Air India and Continental Airlines (0845 607 6760; www.continental.com) from Heathrow and by Continental from Glasgow, Edinburgh, Belfast and Manchester.

Visiting there

Governors Island (001 212 440 2200; www.govisland.com) is open in the summer only: 10am-5pm on Fridays, to 7pm at weekends. The free Governors Island ferry departs regularly from the Battery Maritime Building located at 10 South Street, adjacent to the Staten Island Ferry in Lower Manhattan.

More information

New York City Tourist Office: www.nycgo.com