Arnie Graf: The sage who can see Ed Miliband at No 10

The 69-year-old American inspired Barack Obama's route to the White House. Now he has set his sights on a Labour victory. Jane Merrick meets Arnie Graf

Graf's view of the world changed after a chance experience when he was eighteen (Jason Alden)

For a man who has provided political inspiration to the President of the United States, there is no air of grandeur about Arnie Graf. Perhaps it is his upbringing in New York's Lower East Side, or the fact that, as a community organiser for nearly 50 years, he must have spoken to tens of thousands of what we in Westminster lazily call "ordinary people".

Few outside the political bubble will have heard of the 69-year-old, but he is one of the most influential figures in the circle of Labour leader Ed Miliband. Graf is helping to reinvigorate the party's grassroots. And he can see something that many voters, even Labour supporters, cannot: Miliband as Prime Minister.

So who is the man who could be the key to Labour winning the next election? Graf helped to set up the first "living wage" ordinance – the policy at the heart of Miliband's 2015 manifesto – in Baltimore in the early 1990s; and, in 1986, he was a community organiser when he met a young Barack Obama. After a job as a cleaner, his father became a salesman – a working-class man getting on, as Graf puts it.

But the most dramatic influence on Graf's politics came when he was 18, on a spring evening in 1961 in the bar of University at Buffalo. Graf was a member of the university's intramural basketball team. After an end-of-term match, the team, including one black player called Vernon, went to the college bar. Graf queued for a long time without being served, eventually leaning over to grab the barman's attention. The barman said: "Get rid of the negro and I will serve you." Says Graf, whose Jewish parents had suffered anti-Semitic abuse, said: "My parents always told me, we don't call people names. That is not proper behaviour." When Graf put out his arm again, the barman called him a "negro-lover" and Graf "just lost it". He was thrown out. None of his friends, including his black team-mate, came to his support.

A few months later, Graf saw Vernon eating alone in the cafeteria, and asked him why he had not even made eye contact since the incident. Vernon replied: "I decided you are either ignorant or a fool." Vernon explained that his sister was at the same university, studying to be a lawyer. To pay for college, their mother took two buses every day "to go to a white lady's house to clean it", returned home to cook dinner, before going out to a second job. Their father also had two jobs. Vernon said: "This is what they do, every day except Sunday, so my sister and I can go to university. What would have happened if I reacted? I would have been arrested for disorderly behaviour, then I would have been expelled from school, then I would have broken my parents' hearts. Everything they had done would have been for nothing. I cannot be around people like you."

Graf adds: "I got very teary-eyed. I said: 'I am both ignorant and naive.' This was a picture that stayed with me. It changed me. It made me livid." Graf never saw Vernon again. But by the time he left Buffalo he was channelling his anger at the bar incident into the civil rights movement. He later married Martha, an African-American academic, in 1973 – 40 years ago this month. Graf's work centred on the Industrial Areas Foundation, a Chicago-based community activist group, and through this he met a 25-year-old Obama at a seminar in LA, who spent a week learning about the methods that helped him win the 2008 presidency.

Graf's long-standing interracial marriage interested Obama because that of his own parents, a white mother and an African father, had not survived beyond a few years. "When I met him I didn't see a future president – but that's because I never believed it possible. I tried to recruit Obama as a community organiser; he was very impressive. But by the time he came to me he intended to do a law degree."

Graf met Miliband in December 2010, and was similarly impressed – particularly when the Labour leader told him he had put the living wage in his leadership campaign. Miliband told him he wanted the party to be in touch with the low-paid disenfranchised: "No Democrat had ever said that to me. People on low incomes, they don't see what the heck politics has to do with their life. They need a reason to vote."

Graf carried out a "root-and-branch" review of the party in 2011, and is now working to get Labour's local parties to engage with community groups. But, Graf says, there is something else that Miliband has beyond a strategy to build from the grassroots: growing up in Hampstead with his intellectual parents, the young Miliband not only had ideology, but also developed compassion. Graf says: "He told me 'There was something else in my heart – it was real people doing real things.' " He adds: "I feel I have a connection with him." The majority of voters do not see this, yet, but Graf believes eventually they will.