It is a picture-perfect late-summer evening at the Greek Theatre, an outdoor amphitheatre carved into the thickly wooded hillside at the base of Griffith Park in Los Angeles. The capacity crowd is thrilled at the prospect of hearing a man they regard as a legend, Paul Simon, along with his scintillating seven-piece band. Simon obliges with a set that introduces some songs from his latest album, Surprise, explores the thrilling South African rhythms and vocal harmonies of Graceland, his enduring success from 1986, and offers up lively rearrangements of his beloved old songs - everything from the Simon and Garfunkel hit "Bridge Over Troubled Water" to his signature encore number, "Late in the Evening".
The audience is fired up, the band crackles along with verve and musical precision, and yet some crucial element is missing. As the evening carries on, it becomes apparent that the problem lies with Simon himself. He stands at centre stage, but he never really owns the space. He appears diffident and uncomfortable, rarely cracking a smile and certainly offering no hint of pleasure at being the focus of several thousand adoring fans. Rather, he concentrates on directing the musical activity around him, like an anxious conductor never quite allowing himself to trust the orchestra to live up to his high expectations.
Just once, during a solo performance of the new song "Wartime Prayers", does the evening attain the level of true magic. The song is a mother's lament for what is recognisably the US invasion of Iraq, the melody set against an accompaniment of considerable contrapuntal complexity that Simon pulls off brilliantly on his acoustic guitar. For the one and only time in the evening, he is completely absorbed in the music, without self-consciousness or anxiety. And the audience, clearly moved, is reduced to an awed hush.
When I speak to Simon the next day in his suite at the Bel Air Hotel, he readily acknowledges his unease at being the charismatic figure at the centre of his own show. For him, the pleasure of performing lies elsewhere. "I'll find it a satisfying evening if the band is playing well," he says, "even if the audience is tepid. Even if the audience response is riotous - and we almost had a riot the other night in San Diego - for me it's still how you play. The nuances of our playing change all the time. When the conditions are right and we can command those nuances, then it's a lot of fun."
Even the word "fun" does not fall from Simon's lips with great conviction. (Perhaps because, in that moment, he was feeling less than satisfied with the Greek Theatre set - when I told him how impressed I was with the band's musical flexibility, his face suggested he had to hold back from contradicting me.) He is a shy, serious man, who does not appear to find the creative process any easier as he gets older, who feels, for better or worse, that he is very much ploughing his own musical furrow, and whose conversation frequently hints at disappointments, frustrations, even anger at the world around him.
When Simon was a child, his parents nicknamed him Cardozo after the Supreme Court Justice Benjamin Cardozo who was famous for never smiling. That seemed to give way to a more happy-go-lucky persona during his Simon and Garfunkel period, but now, at the age of 65, he comes across as a singularly intense, melancholy-tinged human being.
That attitude may also be born of disappointments in the past couple of decades. Since Graceland and its Brazilian-inspired follow-up, Rhythm of the Saints, which came out in 1990, Simon has done little to excite either the critics or his fans. A Broadway musical that he worked on for much of the 1990s, The Capeman, was a flop. His last studio album, You're the One, came and went almost unnoticed in 2000.
Now, though, he has a new album that is at last garnering comparisons with the best work of his past. Surprise contains meditations on everything from life after September 11 to a father's love for his daughter. Musically, the most surprising thing about it is that it is the fruit of a collaboration between Simon and Brian Eno - who is credited in the notes for a "sonic landscape" that takes Simon's trademark inventive rhythms and melancholy harmonic progressions and infuses them with an electronica buzz.
Like everything else Simon has done since Graceland, Surprise was written "backwards", in that he started out with the rhythm to each song, then moved on to harmony, added the tune and finished by writing the lyrics. Or, as he put it: "Instead of going in with a song and cutting a backing track, I now cut the backing track and then write the song." He had adopted that approach on a few songs early in his career - "Cecilia", "El Condor Pasa", "Mother and Child Reunion" - but developed it as a working method as "a reaction to a certain process of writing songs - going into the studio with a bunch of musicians and coming out with less than what I had imagined going in".
With each album, Simon aspires to explore something new. Finding a formula and sticking with it, in his view, is to risk turning into "one of those people who repeat anecdotes and forget they've already told you the anecdote - it's an imitation of spontaneity". And so he takes inspiration from anything and everything - Africa, Ravel, or, in this case, Eno - and tries very consciously to bend and break the usual rules of popular music. "I'm interested in changing the song form," he says, "that's what makes me go on."
Thematically, what most stands out about Surprise is its exploration of the times we live in. The opening track, "How Can You Live In The Northeast?", is a very explicit response to the events of September 11, while the song "Outrageous" is a wry commentary on the culture of public anger that has raged since then. "After 9/11," Simon says, "one of the things that artists went through was to ask, what does this say about art? What's your response? I didn't find it easy to respond."
I ask him what he thinks of Theodor Adorno's remark that, after Auschwitz, poetry was no longer possible. "It's not possible for a certain period of time," he says. "But poetry is bigger than Auschwitz. Art is bigger than politics. When an event that traumatic occurs, it poses a challenge to art to absorb it and produce some new form of art. But it's human nature to create."
Simon's personal answer to the conundrum was to realise that he didn't want to put any anger that he felt at the state of the world at the core of his work. He didn't want to be either too pessimistic - "people have enough of that in their own imaginations and their own lives" - or too optimistic if that wasn't the way he felt. "Anything else," he realises, "would be OK with me."
He clearly harbours political anger against the Bush White House - especially over the Iraq war -- and sounded faintly embarrassed when I asked him about the time he was invited there in late 2002, a few months before the Iraq invasion. He was one of a small number of prominent artists to be honoured that year by the Kennedy Center in Washington - a ceremony that by tradition entails a trip to the White House. At first Simon said, with considerable vehemence, that he would not have set foot in the place for any other reason. But then he changed his mind and acknowledged that an invitation from a sitting president is just one of those things you don't turn down. The White House, he says, was "bigger than the occupant, no matter who that is".
When he describes his encounter with Bush, he offers the first real glimmer of humour of the interview. "Bush told me, 'You have no idea how many hours I've spent out jogging with Graceland in my earphones'. Well, that completely changed my opinion," Simon says drily. "I thought, what a great guy."
It would be a mistake to over-read the political content of Surprise, or any of Simon's other work. He remains, at heart, a storyteller and a musical experimenter who sees the lyrics of a song as just one facet - not, emotionally speaking, the most important facet - of a complex artistic enterprise. "When people describe my albums," he says, "they tend to talk about what the words are. But what is going on on a non-verbal level is meant to be a great deal of the pleasure of it. That's its great power. If you have a great melody it can elevate a song into something deeply emotional even with a trite lyric ... It's what is just beneath the surface of the melody and the rhythm. Feeling lost in that space is what gives me pleasure. I don't get so much pleasure being lost in the lyrics."
The collaboration with Eno came about more or less by accident. The two met at a dinner party in London, and Eno invited Simon to come to his studio and play a few demos of his work in progress - at that stage, partially completed versions of a handful of the songs that ended up on Surprise. "He took the sounds and put them through his electronica, and I really liked it immediately," Simon recounts.
Many people thought the pairing was ill-advised. "They either said, 'What is Brian doing messing up Paul?' Or, 'What is Paul doing going into Brian World?'" In the end, they met four times, for five days each, playing with each of the songs and altering its sound in ways that both men found rewarding. "He works fast, which is good for me because I don't," Simon says. "He made shapes I wouldn't normally make... Plus he was a delight to be with. He reads, he's political, he's curious. Those days were a lot of fun."
After two failed marriages (the second a rollercoaster 11-months with the actress Carrie Fisher) Simon is now happily spliced to the singer-songwriter Edie Brickell, with whom he has three children. There are no problems with that other important relationship either, the one with Art Garfunkel. Their partnership ended acrimoniously in 1970, but their reunion tour a couple of years ago was a great success, even if Garfunkel did say that it was the 50th anniversary of their first meeting, only for Simon to interject, deadpan, that it made it the 48th anniversary of when they started arguing.
Simon deserves much credit for continuing to be musically curious and innovative, even at the risk of losing the mass audience - and rapturous critical reception - that was his for the taking for the first two decades of his career. Contemporaries such as Bob Dylan and Neil Young enjoy a greater mythological status by now, and the Rolling Stones have sold more records. Both critically and commercially, Simon might have grounds to feel a little short-changed. Not that the pattern wasn't established a long time ago: the first album of his that was not a multi-million-selling international hit, 1983's Hearts and Bones, has probably held up as well as anything he has done. (Simon calls it "the Paul Simon album that hard-core Paul Simon fans like the best".)
By now, Simon feels as removed from the mass audience of his early hits as some of them evidently feel from him. "What I have on my mind is not what most people have on their mind," he says, sounding more resigned than regretful. He hears all sorts of popular music around his house, thanks to his teenage children and his wife, but he doesn't feel much kinship with what anybody else is doing at the moment. "Most of it I don't like," he admits. "I don't feel I'm in the midst of it any more. I feel I'm in my own place. There's something about that which is both comforting and a little bit lonely - part of getting used to this age."
One thing his age does not portend is retirement. Having explored the possibilities of rhythm for much of his career, Simon is now thinking of starting over from a completely different perspective. "I'd like to investigate harmony," he says. "I haven't begun, so I don't know exactly what that means. Instead of starting with drums and a groove that I like, I would find melody in a harmonic way." Talking about this future ambition, Simon sounded more daunted than enthused - like a weary traveller contemplating a new journey. "Setting up the premise for a 10 to 12 song cycle generated harmonically," he says, "it's not going to be a walk in the park." And, as he well knows, there's no guarantee of the kind of reception he can expect when the journey is over.
Paul Simon's UK tour starts this Friday at The Armadillo, Glasgow, and ends at Birmingham NEC on 11 November. For tickets, see www.livenation.co.uk