Collected ghost stories, by MR James (edited by Darryl Jones)

The phantoms in the library

Quite a few writers exhibit the virtues of their faults. But surely only a small handful of classic authors have bred a florid art from an arid life as Montague Rhodes James did. A century ago, on every Christmas Eve, the arch-conservative Provost of King's College, Cambridge, used to gather favourite students and colleagues around the fire in his lodgings. As the candles flickered, the banter faded and the port glinted in the glass, he would read to them from papers covered with a spidery scrawl.

Today, MR James remains by common consent the greatest author of ghost stories from the British Isles – although, to some connoisseurs, his Irish predecessor Sheridan Le Fanu runs him close. This Halloween, thanks to an outstanding new edition, James aficionados can revisit many of the scariest, craftiest supernatural tales ever told, while envying newcomers their first exposure to the bone-deep shiver of canonical chillers such as "A Warning to the Curious", "Casting the Runes", "Number 13", "The Stalls of Barchester Cathedral" and "'Oh, Whistle, and I'll come to you, my Lad'".

Monty James's circumscribed, even complacent, existence shaped and honed his skill. A clergyman's son, born in 1862, he grew up near the Suffolk coast - and famously made the wind-scoured shingle around Aldeburgh a domain of nerve-shredding dread. Already a precocious scholar at Eton and King's, with medieval manuscripts and catalogues his dusty forte, he smoothly became a fellow of his college, then Dean, then Provost, before a return to lead Eton in 1918 – after the Great War robbed him of a generation of students in a horror far beyond his darkest imaginings. He died in 1936.

This exemplary edition, by Darryl Jones, presents 33 stories in which retribution from beyond the grave erupts with a formless and fathomless intensity. Along with comprehensive notes (these donnish nightmares need them), Jones treats us to judicious scrutiny of the man, and the mind, behind the fright. Reader, you probably would not have taken to Monty – especially if you failed to be born male. Scornfully anti-intellectual, for all his prodigious learning in "apocrypha, ephemera, marginalia", deeply reactionary, nastily misogynistic, firmly celibate, he stood in Cambridge like an ivy-clad brick wall against all progress. Jones shrewdly notes that his younger King's colleague JM Keynes – liberal, reformist, free thinker, free lover – acted as Monty's double, antagonist and nemesis.

The stricken scholars of his tales, reduced to gibbering wrecks by mobile drapery, scratchings in the night or even by a mere hole in a page (the hole that leads to hell), poke their spectacled noses into places they should leave well alone. Curiosity kills the cataloguer. Disturbers of arcane lore secreted in libraries, churches, mazes or burial mounds invite a terrible revenge from the forces of darkness. Let sleeping dogmas lie, we conclude.

Jones teases out the sexual undertones in the work; indeed, ghastly pits, wells or cavities often threaten the unwary bachelor don. Monty was an almost-exact contemporary of Freud, whose essay on "The Uncanny" locates the lure of otherworldly tales in the return of the repressed – the individual drives, and shared beliefs, we pretend to have left behind. The formula fits James like a clammy glove.

Yet the overwhelming power of his yarns could only have stemmed from an imagination that joined a mesmerising precision about details – the inscriptions, the architecture, the topography of terror – with a stubborn naivety over ideas. Too much self-consciousness would have blown the spectres away. In these stories, as undead now as in the Edwardian twilight that spawned them, his lack is our gain.