Shutter Island, Martin Scorsese, 138 mins, (15)

DiCaprio stars in a Fifties Gothic mystery, set in a hospital for the criminally insane, that will leave you baffled but entertained

There may come a point, halfway through watching Shutter Island, when you wonder who – you, or director Martin Scorsese – has taken leave of their senses.

The moment at which this happens, I predict, will be when the rats appear: an image so jaw-droppingly preposterous that some of you will give up on the film altogether, although others will sit tight, telling themselves, "Come on – this is Martin Scorsese. He knows what he's doing, doesn't he?"

There's little doubt that Scorsese knows exactly what he's doing in Shutter Island, and which favourite films inspired him to do it. Adapted by Laeta Kalogridis from Dennis Lehane's novel, Shutter Island is a heady cocktail of genres, not so much a hybrid as a grotesque chimera – part hard-boiled cop story, part locked-room mystery, part Gothic spooker. The premise: cop investigates mental hospital, finds himself gazing deep into his own troubled mind. The year is 1954, and Leonardo DiCaprio – whose pug-like jowliness now recalls such vintage scowlers as Edward G Robinson and John Garfield – is Teddy Daniels, a US marshal visiting the secluded Shutter Island, site of Ashecliffe, a maximum security hospital for the criminally insane. A woman patient has disappeared from her cell; trying to find her, Daniels begins to feel that hospital director Dr Cawley (Ben Kingsley, left, all knowing smiles and pipe smoke) isn't altogether co-operating with his investigation. And when it comes to pressing the cop's buttons, the real expert is sardonic Dr Naehring (Max von Sydow at his most languidly august), who need only speculate that Daniels is a "man of violence" for the cop's nerves to start jangling.

The one thing missing in Ashecliffe is a sign saying, "You don't have to be mad to work/visit/languish here, but it helps." You know from your 1940s /50s films noirs that movie shrinks are prone to be far crazier than their patients: it's the bow ties and European accents that give it away. As for Daniels, his traumas arise from his presence as a US soldier at the liberation of Dachau, and from the death of his wife (Michelle Williams), who appears to him in vividly-evoked dreams, crumbling to ashes in his arms.

As for Ashecliffe's regulars ("only the most dangerous, deranged patients," says a guard, proudly), it's Ward C that holds the A-list. An impenetrable fortress built on the architectural principles of a Piranesi engraving, Ward C is a labyrinth of Gothic darkness, housing a collection of cackling, muttering and physically scarred inmates, one of whom – his briefly glimpsed back covered in religious tattoos – appears to be Scorsese's nod to the villain of his own 1991 thriller Cape Fear.

Despite its references to modern history – to the Holocaust, the House Un-American Activities Committee, and the Cold War – Shutter Island bears tenuous relation to the real world. Rather, the film seems to be made from echoes of other works: specifically, the 1940s chillers made by the legendary producer Val Lewton (Isle of the Dead, Bedlam) and Samuel Fuller's asylum thriller Shock Corridor. And watch for the shower head from Psycho – a rare example of a star cameo from a plumbing attachment.

Some viewers will feel that Shutter Island's presentation of mental illness is sensationalist and retrogressive. In fact, the film appears to articulate some sort of argument about psychiatry, its disreputable history and its representation by Hollywood. But this is not what interests Scorsese most. What he's really concerned with in Shutter Island is cinema – the consensual madness that we yield to when we enter the darkened auditorium. The island itself, with its projector-like lighthouse, is a movie theatre, and so is Daniels' mind. Cinema and psychoanalysis share some key terms, after all: "projection", "screen".... Oh, and a shutter is part of a camera.

As hokum, Shutter Island doesn't quite transcend its pulpy origins as The Shining did (like Kubrick's film, Scorsese's features a stark modern classical soundtrack, including Ligeti, Penderecki and other nerve-shredders). The film is executed with panache; it's stormily shot by Robert Richardson, and a superb cast includes Emily Mortimer, Patricia Clarkson and Jackie Earle Haley. And while DiCaprio is very compelling as the frazzled hero, the scene-stealer is Mark Ruffalo as Daniels' deputy: he may spend his time looking affably bemused, but in his body language (the way he wears an overcoat, holds a cigarette), he insouciantly channels the four-square masculinity of such 1940s/50s stars as Dana Andrews or Fred MacMurray.

The biggest problem is the use of Dachau as a back story, handled in a way that makes something beautiful and dreamlike out of images that no film-maker should exploit for metaphorical resonance. It's a lapse of taste that Scorsese should have risen above.

This is a serious false note in what otherwise is a very arresting oddity. Shutter Island is one of those hyper-devious mind-game movies that are so busy pulling out rug after rug from under your feet that you eventually find yourself wondering whether the makers remembered to provide any floorboards. You suspect that a director of less exalted auteur repute – David Fincher, say – might have carried the film off with less fuss, and more clear-cut conviction. Even so, Shutter Island, flawed as it is, is more audacious, challenging and downright entertaining than most Hollywood genre movies today. And many times more delirious.

Next Week:

John Walsh investigates Seventies-style retro-chiller House of the Devil