On the shortlist for this year's "Booker of Bookers" poll, the darkest horse galloped hard: JG Farrell's The Siege of Krishnapur. The 1973 prize-winner, from the Liverpool Irish novelist who drowned while fishing in Bantry Bay in 1979, has won a legion of new readers. Once patronised as a maverick or a throwback, Farrell now looks ahead of his time. In his character-rich, adventure-stuffed, satire-strewn sagas, he cheerfully kicked down the walls between "literary" and "genre" fiction.
Behind Farrell's return to favour lies a hunger, among readers of fiction and certainly among anxious publishers, for the blockbuster with brains. "Literary fiction" in its familiar formats has slumped deep into recession as first-rate writers struggle to keep a toehold in a hostile marketplace. At the bottom of the barrel, bog-standard escapism and ghosted celebrity fluff prompted the agent Clare Alexander, a few months ago, to denounce Britain's book charts as "the stupidest bestseller list in the world". As of late July 2008, the horizon looks slightly sunnier: Rose Tremain has been outselling Russell Brand. But the overall climate for fiction has darkened even more.
So the hunt accelerates for skilful popular fiction that can slip smoothly into a beach-bag or a deck-chair without insulting its readers. Henry Porter's political thrillers have updated the murky geopolitics of John Le Carré to a post-Cold War world where Balkan civilians are massacred, Srebrenica-style (A Spy's Life) and terrorist suspects "rendered" for torture (Empire State). He tells me that he's "unimpressed by the rigid distinctions between literary and popular fiction – and the attempts to patrol them – because the readers I encounter at literary festivals flit from Robert Harris, to Dickens, to Jane Austen and back to Rachel Johnson and Joseph O'Neill [of Netherland fame] ... Readers are smart. They don't need to be told what is good and what is bad fiction".
Some pop genres have traditionally held a licence to thrill, to divert and to exercise the mind. Crime has never forfeited that status since the days of Conan Doyle. From HG Wells to Iain M Banks, science fiction led the way in fusing high-grade ideas with mass-market appeal. Patrick O'Brian's naval yarns brought the sweep of Proust to shipboard adventure. And, since Greene and Le Carré, espionage claims its touch of class.
Now, no formula seems too tainted for a careful restoring hand. Robert Harris expertly dusted off the toga tale with his Roman intrigues Pompeii and Imperium. Kate Mosse proved, in her vastly successful Labyrinth and Sepulchre, that ancient puzzles hidden amid the stones of Old Europe can come to light free of Dan Brown-level banalities. Mosse believes that "there are lots of readers who want to have a rollicking good story – plot, characters, setting – but who also want to have a sense of real history and real place." In 1980, the medieval mystery that first detoxified this brand came – almost too neatly – from an iconic European thinker who had written much about the fuzzy borders between elite and mass culture: The Name of the Rose by Umberto Eco. Here was one intellectual who proved that he could walk the walk as much as talk the talk.
From the Arabian Nights to Don Quixote, the origins of the novel lie in genre storytelling. By reverting to narrative-driven adventure, mystery or suspense, today's brighter bestsellers have leapfrogged the mere 150 years of the self-defined "literary novel" that began with Flaubert. As Kate Mosse argues, "Nowadays, books we think of as 'classics'" – such as Treasure Island or Robinson Crusoe – "in their day were all the equivalent of beach reads". In any case, intelligent popular fiction is an exacting craft. To Porter, "Writing a thriller is a complicated business at the best of times, and to make it real and relevant, at the same time as keeping the plot moving and the characters breathing, is bloody hard work."
Almost a century ago, a respectful but tetchy correspondence between HG Wells and Henry James set the terms of a debate that rumbles to this day. James championed uncompromising "art" fiction as the royal road to truth: "It is art that makes life, makes interest, makes importance". Wells spoke up for nourishing entertainment that taught via enchantment. James won in the academy; Wells at the tills. Their battle still rages. Mosse comments: "Most readers want, especially on the beach, something that has fast pace, strong story, strong characters, but with depth to it too... the sense that I've learned something at the same time as being entertained".
Novelists who seek broad-spectrum popularity but shun the lure of pulp need to know (and feel) the pull of both sides in the Wells-James quarrel. But if the "literary" corner of the fiction shelves look dusty and unloved, then even more ambitious authors will take a chance with the sun-lounger brigade. Many high-flying angels – as well as barrel-scraping devils – will seek to catch your attention between the airport-bookstore dumpbin and the deep blue sea.
Ten of the hottest books this summer
Sashenka, by Simon Montefiore
In his first novel, the acclaimed historian of Russia sweeps the brittle high society of pre-revolutionary St Petersburg, the terror-chilled jails of Stalin's purges and the secrets of 1990s Moscow archives into a tragic panorama. This family saga captures both the epic travails of a Bolshevik elite that fell from grace, and also – ambitiously – the enigmas of historical knowledge itself.
The Ten-Year Nap, by Meg Wolitzer
(Chatto & Windus, £12.99)
What comes after chick-lit? Mum-lit, perhaps - but tales of hyper-active kids, moribund marriages and the career opportunities that got away will seldom match Wolitzer's wit, bite and schmaltz-free sympathy. As in a reversed-out riposte to Sex and the City, her quartet of New York friends shape up to the outcome of their choices and chances, a decade after motherhood.
Palace Council, by Stephen L Carter
(Jonathan Cape, £12.99)
Yale law professor and weaver of history-haunted thrillers, Carter has dodged all the pigeonholes that snare African-American novelists. But the troubled rise of the black middle-class is never far from his intricate plotting. Here, the end of the Harlem Renaissance meets Cold War conspiracy and Vietnam-era activism in a richly involving brew.
The Return, by Victoria Hislop
Shifting the formula of her debut The Island from Greece to Spain, Hislop wraps another British innocent-abroad envelope around a meaty – and moving – slice of modern history. Via a café in Granada, her dance-crazy heroine discovers the Spanish Civil War – evoked through the trials, on the home front and the front line alike, of the warmly drawn Ramirez family.
Revelation, by CJ Sansom
Sansom's superb Tudor mysteries do more than marry period drama with gory crimes and smart detection. He respects history while forging links to today. Here, religious cults in 1540s London bring his Shardlake into conflict with jihadis in the English past.
Devil May Care, by Sebastian Faulks
You might say that Faulks's James Bond novel made a risky mission harder. It refuses to customise the franchise and, instead, aims for a note-perfect pastiche of a late-1960s Ian Fleming yarn. Having boxed himself in a bit, Faulks wangles an escape – 007 style – by virtue of the pleasure and panache of an expert period piece, as Bond tussles with a memorable adversary: Julius Gorner.
The Pirate's Daughter, by Margaret Cezair-Thompson
Errol Flynn did land in Jamaica to film in 1946, but here his swashbuckling arrival launches a fiction that sails into harbours that Hollywood would never dare to visit. Racial and social tensions cloud the Caribbean idyll: Cezair-Thompson mixes the romance and reality of her native island into a cocktail that blends the exuberant sweetness of its fantasy with the sharp tang of truth.
The Outcast, by Sadie Jones
If McEwan lies behind this finely managed tale of disgrace and redemption, then surely so does Dostoyevsky – the author's literary inspiration. Back from an actual jail into the shackles of class, family and past trauma in 1950s Surrey, young Lewis becomes a catalyst for truth in a land of lies. This green-belt hell exerts a politely chilling grip.
The Spies Of Warsaw, by Alan Furst
Over 10 novels, Furst has stylishly revived the mid-century intrigue of middle-European shadows and fog. Warsaw in the late 1930s provides his latest battleground for spies and lies. French and German agents play their deadly game of chess as war looms. Furst's uncanny gift for place and period lift his city, and its dubious cast of characters, well above the espionage norm.
Sea of Poppies, by Amitav Ghosh
(John Murray, £18.99)
Ghosh has a fair wind for his 19th-century maritime trilogy. In this first part, the Ibis sets sail for the Opium Wars. The ship's motley mix of races, trades and lingos enrich the texture of a well-researched romp. Colonial conquest in Asia paints a vivid background canvas.