It is one of the most maligned inventions in the history of music; the greatest signifier of rock star hubris. The very mention of it is likely to provoke snorts of derision among the most open-minded rock fans, conjuring images of quadruple-gatefold album sleeves, songs that go on for days and shaggy-haired rockers prattling on about unicorns, goblins and dystopian futures. I am, of course, talking about the concept album. But if you thought this absurd creation had gone out with capes and dry ice, think again, because a generation of musicians is doing its damnedest to bring it back to life.
Following a scattering of concept albums last year – including Jim Noir's eponymous LP told from the perspective of an astronaut looking down from space, and Neon Neon's Stainless Style, about the rise and fall of the car pioneer John DeLorean – rock's big guns are being steadily seduced by its unwieldy charms.
Mastodon, The Decemberists and Neil Hannon (as The Duckworth Lewis Method) have all released concept albums this year. Muse recently offered up The Resistance, a quasi-concept piece referencing Orwell's Nineteen Eighty-Four that shoehorns a love story into a futuristic world of political and ideological subjugation. Next week sees the release of Embryonic, the new Flaming Lips album arranged around the zodiac.
It certainly seems surprising in an era where consumers have broadly pitched themselves in the middle of road and musical ambition is, more often that not, greeted with suspicion. But, then again, let's consider the bands in question. Muse are stadium rock giants known for their cod-operatic preposterousness and for whom song titles such as "Exogenesis: Symphony" prompt no flushes of self-consciousness whatsoever. And then there's the Flaming Lips, a band who once orchestrated a four-CD album that was designed to be played simultaneously. For these bands, vaulting ambition is all in a day's work.
The concept album was originally defined as a long-player where the songs were based on one dramatic idea – but the term is subjective. There is a difference between an album based on a theme and one defined by a narrative. No wonder, then, that the identity of the first concept LP depends on who you ask. The Beatles' Sgt Pepper's Lonely Hearts Club Band is widely considered to be among the first. What's clear is that before prog rockers such as Yes and Jethro Tull hijacked the format in the Seventies, concept albums were straightforward affairs. One of the earliest narrative albums may be Woody Guthrie's Dust Bowl Ballads of 1940, a collection of songs about the dust storms that ravaged the Great Plains in the Thirties. Frank Sinatra's In the Wee Small Hours, released in 1955, was a cycle of ballads telling the story of a man whose lover had left him and his reflections in the early hours of the morning.
Brian Wilson certainly brought big ideas to the Beach Boys. Pet Sounds in 1966 brought new depth and complex arrangements to the themes of love and loneliness. It is regarded as one of the great concept albums, even though it wasn't intended as a narrative. Wilson had even bigger plans; the follow-up, Smile, was meant to be his "teenage symphony to God". But it all proved too much and, after trying to burn down his studio, Wilson retired to his bed, leaving the project languishing in a vault for 40 years.
More conceptual efforts followed in the Sixties, with Frank Zappa, The Kinks and Pretty Things' each releasing concept albums. By the end of the decade the format had even developed a sub-genre – the rock opera – typified by The Who's Tommy and, in the Seventies, Quadrophenia. Pink Floyd later embraced rock-opera with The Wall, by which time the concept album had reached fantastical heights of ambition as its creators turned to literature and legend for inspiration, and sought to bring classical music into their compositions. This new school of conceptualists came with outlandish outfits, impenetrable lyrics, choirs, orchestras and great gusts of dry ice. Chief perpetrators were Yes, whose Tales From Topographic Oceans was the apogee of prog excess and came with a concept – something to do with Paramahansa Yogananda's Autobiography of a Yogi – that even the fans struggled to unravel.
There are more redeeming examples, such as Genesis's The Lamb Lies Down On Broadway, which drew on folklore, the Bible and classic literature as it told the tale of a New York hustler. The Rise and Fall of Ziggy Stardust and The Spiders From Mars, on which David Bowie became an androgynous alien rock star, is a classic.
By the Eighties, concept albums seemed to have been consigned to the bargain bucket of rock extravagance. It's perhaps no coincidence that they fell out of favour at the same time as the emergence of MTV, which was all about the chart-busting single – not albums. Pop hooks and slick visuals were in; grand conceits were out.
Throughout the Nineties, concept albums remained thin on the ground, although there were exceptions such as Elvis Costello's The Juliet Letters and Liz Phair's Exile In Guyville, the latter a response to the Stones' Exile on Main St. Since then we've seen The Streets aka Mike Skinner's 2004 album A Grand Don't Come For Free, centred on a narrative about the loss of £1,000, and Green Day's epic 17-track, three-act, post 9/11 rock opera American Idiot, one of their biggest-selling efforts yet.
The concept album clearly has its limitations, not least the possibility that no one will get what it's all about, and perhaps this accounts for its dwindling popularity. For all its creative freedoms, it offers an unwieldy method of showing off your musical prowess. So why are musicians suddenly trying to rehabilitate this totem of rock star folly?
One theory is it's a result of the changing ways we are listening to music. Gone are the days when you had to buy a whole album to hear the one song you actually wanted. With digital downloading, fans can pick the tracks they want and ignore the ones they don't. When many of us listen to our iPods on shuffle, there is dogged sense in making albums that are difficult to listen to in three-minute bursts. It's possible that artists are trying to breathe new life into a dying format and discovering the art of storytelling in the process.
Part of our cynicism when it comes to concept albums probably lies in a collective recoiling at the notion of pop stars – by definition shallow, fly-by-night types – trying to be clever. A similar type of snobbery afflicts the music industry that, notorious for underestimating its consumers, assumes short attention spans among music listeners and shies away from complex formats and ideas.
So, perhaps it's time to stop frowning on musicians aiming a little higher than the three-minute pop song. After all, what would become of our great novelists if they were confined to producing novellas, or film-makers reduced to churning out shorts? Why shouldn't ambition be rewarded, not thwarted?
Muse's 'The Resistance' (Warners) is out now. The Flaming Lips' 'Embryonic' (Warners) is released on 12 October