Omega Minor, By Paul Verhaeghen

A blast of nuclear fiction

It is hard to imagine Omega Minor, Paul Verhaeghen's extraordinary new novel, having the same success in England as it has enjoyed in Germany, the Netherlands and the author's native Belgium. Indeed, it seems likely that the author has translated the book himself not as a display of his polymath abilities but because he might have found it hard to find another translator prepared to take on a 700-page novel about cognitive psychology, quantum physics, Nazis and Neo-Nazis. It would be philistine not to admire the sheer ambition of the book, especially when the market for serious fiction is under endless assault, but the author has a number of quirks that may alienate some readers. Foremost is a bizarre fixation with ejaculation, prompting phrases such as "pearly liquid", "creamy harvest", "frothy broth" and, most imaginatively, "an acrobatic snake snapping at – but missing – its own tail". There are dozens more.

Verhaeghen is an associate professor at the Georgia Institute of Technology, and one of the novel's many triumphs is his vivid depiction of academia, a milieu few authors can make dramatic. Most of the hero's research students have one ambition: to get from European universities to the promised land of America. But Donatella, a physicist hunting for dark matter, has her eyes on the Nobel Prize.

As she explains to Paul Andermans, a Dutch cognitive psychologist, there is a major discrepancy between the mass of visible matter in the universe and the universe's total mass. Einstein created the parameter Omega to define the ratio between the actual density of the universe and the density required to keep it from imploding, before dismissing this idea as the biggest blunder of his life.

Donatella believes Einstein was wrong to retract the notion and that investigation of Omega will tell us what will happen to the universe. She also believes that something is wrong with Omega. Donatella thinks it is tied to the total amount of mass in the universe. Some physicists claim that the missing mass is hidden in MACHOS (massive compact halos objects); others believe they are in WIMPS (weakly interacting massive particles). Some scientists think WIMPS are made out of neutrinos, particles so tiny and fast they shoot through matter, but Donatella believes the missing mass is made of magnetic monopoles, a particle so rare nobody has observed it.

This theory is at the centre of the novel's argument: a cosmological mystery that drives most of the intellectual conversation. The death of the universe has always been a question that concerned serious novelists (particularly Martin Amis), but too often it can be used as a justification for depicting human degradation. There is plenty of unpleasant violence in Omega Minor, but it is elevated by the fact that the characters are searching for evidence that we don't live in a dying universe. There's a lightness to Verhaeghen's depiction of the interaction between Donatella and Paul that counterbalances his desire to explore the 20th-century's worst atrocities.

Among authors of big novels that address science, technology and psychology, Verhaeghen is closer to William T Vollmann, Don DeLillo and the underrated British author James Flint than Thomas Pynchon or David Foster Wallace. He has almost no interest in popular culture and his conspiracy theories tend to focus on historical injustice rather than sinister cabals. Although it is always entertaining, and rarely heavy going, there is nothing whimsical about this book. He doesn't skimp on character detail, and is as good as inhabiting the imagination of a woman sexually attracted to neo-Nazis as in depicting competition among aspiring intellectuals.

Equally impressive is his depiction of pre-Second World War Berlin, the wartime years and the aftermath. So many authors have already explored similar territory: it's extraordinary that Verhaeghen manages to make his survivor's tale seem original. The memories of Jozef de Heer make up the most compelling sections of the book, and as with the contemporary narratives, it is the accumulation of telling details that makes the prose so vivid. His vision of Berlin nightlife is beautifully rendered. That this narrative has a predictable twist doesn't seriously weaken what comes before.

Omega Minor is undoubtedly a curate's egg, but few recent novels rival its richness. And there is something admirable about an author who challenges not just the structural limitations of the novel, but also the limitations of our understanding of the universe. For all its flaws, this is an uncommonly intellectually stretching- and satisfying - experience.

Matt Thorne's latest novel is 'Cherry' (Phoenix)

Dalkey Archive Press £9.99 (691pp) (free p&p) from 0870 079 8897