Up close and personal with the new, all-American anti-heroes

Noah Baumbach's new film, Greenberg, starring Ben Stiller in his first serious role, is the latest in a crop of sour, forensic family movies from the US that refuse to toe the Hollywood line. Geoffrey Macnab reports

It is ironic that Noah Baumbach's Greenberg is being released in British cinemas so soon after the death of Dennis Hopper. Late Sixties and early Seventies US cinema – with Hopper, Peter Fonda, Hal Ashby and Bob Rafelson in the vanguard – was full of stories about disaffected anti-heroes chafing against the hypocrisy and bad faith of their parents' generation. Roger Greenberg (Ben Stiller), the very sullen protagonist of Baumbach's new film, seems at first glance little different from some of the equally disgruntled characters Jack Nicholson used to play in films like Five Easy Pieces and The Last Detail. He is angry at the course his life has taken. An East-Coaster, he is disdainful about the world he encounters in LA when he comes to house-sit for his brother. Like Nicholson in Five Easy Pieces, he is combustible. The smallest provocation can push him into a rage.

However, Baumbach's film is a measure of how much has changed since the Easy Rider era. Rebelliousness has given way to apathy and boredom. Film-makers' gazes have turned inward. There is a defeated quality to Roger Greenberg that the alienated outsiders from Rafelson and Dennis Hopper films never had.

Like his contemporary and friend Wes Anderson (whose Fantastic Mr Fox he co-scripted), Noah Baumbach is more of a satirist than an insurrectionist. He makes quizzical, well-observed and very intimate dramas about middle-class American life. His subject matter is dysfunctional families. His Greenberg is one of a number of recent US films anatomising American family life. This is a sub-genre that used to seem a very European preserve. American cinema didn't have film-makers like Mike Leigh or Ingmar Bergman, probing away at the minutiae of domestic life or exploring social and class embarrassment. When Hollywood made films about unhappy marriages (for example, Who's Afraid of Virginia Woolf?), the spouses went at each other as if they were contestants in a heavyweight boxing bout. Family melodramas, from The Best Years of Our Lives to Terms of Endearment, were likewise on an epic scale. They didn't pay attention to the tiny tics of behaviour that a director like Leigh might home in on.

In the US indie era, there have been many more films prepared to look at the underbelly of family life. Jonathan Demme's Rachel Getting Married, Baumbach's Margot's Wedding, Anderson's The Royal Tenenbaums, Phil Morrison's Junebug and so-called "mumblecore" dramas like Andrew Bujalski's Beeswax are among the movies that have concentrated on aspects of family life at its rawest and most contradictory. What these films all have in common is their willingness to depict frustration and disappointment. The characters the films portray vary from being insecure to downright obnoxious but the film-makers don't judge them. Their heroism, if it can be defined as such, consists in soldiering on.

Baumbach is famously well-connected. His mother is the former Village Voice film critic Georgia Brown. His father is the novelist and film critic, Jonathan Baumbach. He is married to the actress Jennifer Jason Leigh (who appears in Greenberg.) He writes humorous columns for The New Yorker and has contributed to Saturday Night Live. Ben Stiller is one of his best friends. LCD Soundsystem write his soundtracks.

Given his background, detractors have sometimes struggled to take Baumbach's films about domestic angst seriously. "You look at Noah Baumbach's work and you see he's an asshole," a prominent New York critic said of the director in a much-quoted interview. Wes Anderson's studiedly cool quirkiness has likewise antagonised certain reviewers, who've questioned how so detached a film-maker can portray effectively the tensions between siblings or the trauma of bereavement. The references to the novels of JD Salinger (in The Royal Tenenbaums) or the movies of Satyajit Ray (in The Darjeeling Limited) can seem gratingly self-conscious.

The humour in both directors' movies is disarming. Audiences who've long been looking for the next Woody Allen aren't sure just how to take Baumbach and Anderson's films. On one level, the two directors can seem like preppy, contemporary versions of Allen. They have his flair for observational comedy and one-liners. They're from a similar cultural milieu. Watching Greenberg, with its ironic asides about life in Los Angeles, it's easy to be reminded of Allen's comic sideswipes about West Coast life in Annie Hall. Florence (Greta Gerwig), has some of the scattiness you find in Allen's heroines.

Baumbach's films have become progressively sourer. His 2005 feature, The Squid and the Whale, was about divorce but dealt with bleak subject matter in charming and very funny fashion. In particular, Jeff Daniels's philandering novelist was a memorable comic creation: a hirsute, selfish but engagingly ridiculous figure whom you couldn't take against, however shabbily he treated his wife and kids. The parents' antics were seen from the perspective of their adolescent sons, who had an Adrian Mole-like naivety about them.

Margot at the Wedding (2007) had an altogether more bitter taste. Nicole Kidman played a successful writer attending the marriage of her estranged and thoroughly chaotic sister (Jennifer Jason Leigh) to an unemployed would-be musician (Jack Black). Baumbach goes out of his way to accentuate the loathing that the sisters feel for one another. Kidman has never played a less sympathetic character than the duplicitous and bitchy Margot. Jack Black, normally the most engaging and likable of screen presences, has rarely seemed so needy or pathetic.

In Greenberg, likewise, Baumbach pushes audiences' tolerance for his characters close to breaking point. In the title role, Ben Stiller is very obnoxious indeed. We're so used to seeing Stiller in comic roles that we want to like him. We end up waiting for punchlines and redemptive comic moments that never come.

In European cinema, we are more accustomed to intimate family dramas with coruscating levels of viciousness. In Ingmar Bergman films, from Scenes From a Marriage (1973) to Saraband (2003), there are moments in which bickering spouses or fathers and sons treat one another with utter brutality, homing in on emotional and sexual insecurities. There is nothing as extreme as this in Greenberg, but it is still startling to see a US film starring Stiller that is quite so bleak in tone. Not even the presence of Rhys Ifans as Baumbach's "best friend" leavens matters much. Ifans's character is a recovering alcoholic with his own demons.

There is humour throughout the film, but it is invariably to do with embarrassment and misunderstanding. Take, for example, the excruciating sex scene in which Florence seems utterly uninterested in Greenberg's attempts to arouse her.

Neither Margot at the Wedding nor Greenberg have been box-office hits in the US. And films like Junebug or Beeswax have not been big commercial successes. There is a self-consciousness and introspectiveness about this style of film-making which certainly wasn't there a generation ago, when film-makers really thought they could take over the Hollywood system from within. Nonetheless, these films have an emotional honesty which glib US comedies and romantic melodramas inevitably lack. They consistently challenge audiences to have compassion for truculent, awkward characters whose stories mainstream Hollywood still has no interest at all in telling.

'Greenberg' is released on 11 June