Martha Marcy May Marlene (15) / Carnage (15) (4/5, 1/5)

Starring: Elizabeth Olsen, Sarah Paulson, John Hawkes, Hugh Dancy / Jodie Foster, Kate Winslet, Christoph Waltz, John C Reill

Acompelling wave of dread gathers during the tense psychodrama Martha Marcy May Marlene, though for most of its length we can't be sure from which direction it's coming. By the time it breaks over us the film has ended, and we leave it feeling pretty freaked out. It is remarkable for two debuts, Sean Durkin writing and directing his first feature, and Elizabeth Olsen starring in hers. In their different ways they have set themselves a blazing standard, and deserve all the plaudits coming to them.

Olsen plays Martha, a 20-ish woman who's been living in a farmhouse commune in rural upstate New York. Something has happened to her, because she's bolting through the woods and looking for a hide-out. She calls her older sister, Lucy (Sarah Paulson) whom she hasn't seen in years, and finds refuge with her and her new husband, Ted (Hugh Dancy), at their opulent lakeside house in leafy Connecticut. The reasons for their estrangement are left vague, though Lucy's unarticulated guilt over her kid sister and Martha's unworldly habits – carefree nudity, slipping into the marital bedroom mid-intercourse – will not make this family reunion any easier.

Martha, it transpires, has had trouble with another sort of "family". Interleaved with her present sanctuary are flashbacks to her life at the farmhouse, where a seeming idyll of young people working the land and eating communal suppers was dominated by their leader, Patrick (John Hawkes). "Let us in," he asks Martha, "we want to help you," which offer isn't nearly as generous as it sounds. First off, he renames her Marcy May, subtly undermining her identity; he grooms her for a "cleansing" ceremony that is basically a drugged rape; then he serenades her with a campfire song on his guitar, as if to seal his possession. These and other memories grip her psyche like weeds in a dark pond.

The extensive information the film withholds will frustrate some viewers, but nobody should feel shortchanged by the superb performances. The 22-year-old Olsen, younger sister of the famous twins, has a fine, open face that holds the camera's gaze, and conveys through it a desolate sense of confusion. "What is the right way to live?" is a question that haunts her: neither the spiralling moral chaos of the farm commune nor the solid bourgeois comforts of her sister's life seem the right answer. As Lucy, Paulson also plays confused, only in a more impatient, uptight manner; Martha won't give anything away about her recent past, so Lucy can only speculate on the weird and fearful creature her sister has become. The key to it all is Patrick, played with a frightening fox-eyed watchfulness by Hawkes, rather like his Teardrop in Winter's Bone opposite Jennifer Lawrence, but more sinister in his coercive potency: he knows how to charm these emotionally vulnerable young women, and thus how to bind them to him. If the Devil ever gets his own biopic this guy ought to be a shoo-in for the part.

The one word that's not mentioned in the film is "cult", though you can't watch it without thinking of Manson or Koresh and their devoted acolytes. The carnal practices that rule at the farm, licensed under the weasel word "sharing", are creepy enough. What the story seems to confirm, in the maddeningly ambiguous final moments, is that Patrick has taken his father-figure role beyond sexual opportunism into ritual. This cult is a death cult.

Carnage is a satire about an upper-middle, cosmopolitan elite who believe themselves to be smart and liberal – and it's designed for the very same people. Roman Polanski, who's made some memorable movies about claustrophobia (Repulsion, Rosemary's Baby, The Tenant), is adapting the stage play God of Carnage by Yasmina Reza, about two married couples unable to escape one another in a swanky Brooklyn apartment. The only advantage to the film is that you don't have to listen to a theatre audience sniggering at its barbs.

The couples are gathered to discuss an adolescent spat that got nasty. The teenage son of Penelope (Jodie Foster) and Michael (John C Reilly) lost a tooth in a fight with the son of Nancy (Kate Winslet) and Alan (Christoph Waltz), who come to make reparation but get into a separate argument about Michael's abandonment of a pet hamster on the street outside. How could he? Thus is established the rhythm of the piece, appeasing and conciliatory to begin, then aggressive and accusatory, and back again. Each time the encounter seems about to end – they get as far as the building's elevator – they are dragged back into the apartment for coffee and cobbler, or scotch and cigars. And on it goes.

"We're all decent people," says Michael, except that they're not, they're prickly and defensive and smug. Reza's point is that beneath the veneer of civility our human instinct is geared towards hostility and antagonism – carnage, in a word. Whatever the truth of that, the set-up is false: these people would be out of the door and bitching in private, not shoving their grievances in one another's face. Alan, a slick corporate lawyer, can barely keep his mind on their argument anyway, forever yammering to the office on his mobile phone.

The four actors plug away at the awful (translated) dialogue, gamely trying to whip up a froth from this annoying and implausible situation. Foster suffers the worst, her mean slot of a mouth nearly as ridiculous as her character's occasional stabs at piety ("We all have to be collectively concerned"). You honestly wonder what she could have seen in the part to enjoy, apart from trying to thump Reilly's lummox of a husband.

The whole affair is so airless, and joyless, that you soon resent every minute it's detaining you. That hamster was well out of it.