Will it be fourth time lucky for Julian Barnes?

Perennial literary bridesmaid makes Booker list – but bookies' favourite misses out

His latest novel is epic in its subject matter – the life and death of a First World War poet – and in its size but The Stranger's Child by Alan Hollinghurst appeared to lack the heft to impress judges, who dropped the former bookies' favourite from the shortlist for the Man Booker Prize for Fiction.

Hollinghurst's book, described as "a classic" by reviewers and originally the favourite to win after it was longlisted in July, was the surprise omission from this year's shortlist for the £50,000 award, announced yesterday.

Julian Barnes, who has previously been shortlisted three times but never won, is now front-runner to pick up the accolade, for his "short novel" The Sense of an Ending, about a middle-aged man reassessing his early life.

"We had a long discussion about all the books but eventually we agreed on these six," said judging chair Dame Stella Rimington. "The Hollinghurst we took very seriously but it lost." She said there were "no hidden agendas".

Fellow judge, journalist Matthew d'Ancona, said: "The congratulations we extended to the entire longlist still hold. It wasn't the exclusion of a particular book, we had to narrow the longlist."

There are two first-time novelists on the six-strong list of finalists, Stephen Kelman, for Pigeon English, and A. D. Miller, for Snowdrops. D'Ancona said that Kelman's book, whose characters inhabit Peckham gangland culture, was "a form of grim prophecy". "There was an accidental topicality about it," he added. "It has the capacity to endure. It's a very significant book". Snowdrops, meanwhile, is a psychological thriller set in Moscow. "He's created characters which typify an aspect of Moscow today," said Dame Stella.

Patrick DeWitt's The Sisters Brothers, the off-beat story of two outlaw siblings wreaking havoc during the 1850s Californian Gold Rush, is the first "Western" ever shortlisted for the award. Also making the cut were Carol Birch, for Jamrach's Menagerie, based on the true story of the sinking of the whaleship the Essex in 1820, and Esi Edugyan for Half Blood Blues, revolving around a mixed-race jazz trumpeter in 1930s Berlin.

DeWitt's publisher, Granta, joins three other small independent publishers on the list: Canongate, Atlantic and Serpent's Tail. Jonathan Cape is the only "conglomerate publisher" represented, for The Sense of an Ending.

The judges emphasised that their assessment was "subjective". "For goodness sake, don't think these are the only books to read this year. There were 138 books submitted. It's our own judgement," said author and judge Susan Hill."We were sad to leave some excellent books behind," added Dame Stella. The winner will be announced on 18 October.

Six of the best: The Booker shortlist

Carol Birch Jamrach's Menagerie (Canongate Books)

Author of nine previous novels including Scapegallows and Turn Again Home – longlisted for the Booker Prize in 2003. Jamrach's Menagerie was inspired by the real-life sinking of a whale ship in 19th-century England and charts the fortunes of the charismatic Charles Jamrach, a leading wild animal dealer.

Julian Barnes The Sense of an Ending (Jonathan Cape – Random House)

The fourth novel by Barnes to be shortlisted for the Booker prize, yet he has never taken home the award. Focusing on a group of old school friends, this novella is a meditation on the themes of ageing, memory and regret.

AD Miller Snowdrops (Atlantic)

The debut novel by one-time Moscow correspondent of The Economist, Snowdrops is a tense psychological thriller that unfolds over the course of a brutal Moscow winter. The book tells the story of a young Englishman caught up in a corrupt property deal in new Russia.

Esi Edugyan Half Blood Blues (Serpent's Tail)

The second Canadian on the shortlist. Esi Edugyan's second novel is partly set in the aftermath of the fall of Paris in 1940, following the fate of an arrested German, a black jazz trumpeter called Hieronymous Falk, and in Berlin 50 years later as his bandmates retrace his steps.

Patrick deWitt The Sisters Brothers (Granta)

The second novel by the Canadian-born, Oregan-based author. The Sisters Brothers is a darkly comic western about two outlaw brothers who are hired killers – one reluctant, the other more gung-ho – as they reconcile their relationship during the west coast gold rush in 1851.

Stephen Kelman Pigeon English (Bloomsbury)

After finishing his degree, Stephen Kelman worked as a warehouse operative, a careworker, and in marketing and local government administration, until finally deciding to write in 2005. His first novel is written from the perspective of a seven-year-old Ghanaian boy caught up in Peckham's gangland.