The other day, I met a young man in his twenties called Jim. He is civic-minded, so decided to volunteer for a local charity. The first two he contacted said they would be in touch, but were so under-resourced they never called back. The third rejected his offer, saying there were no opportunities. The fourth was interested, but told him he must undergo Criminal Records Bureau checks, which would last weeks. So Jim gave up his attempt to help other people.

My friend Ash works for a charity so dependent on government funding that he is terrified to say anything in public in case it loses its support. Then there is Sally, who got involved in local politics but gave up because there was so much bureaucracy it was impossible to change anything. Now she doesn't even vote.

Here are three young people living near me in Tower Hamlets, the most disadvantaged borough in the country, who want to help and make a difference to the people of east London – but who have joined the swelling ranks of the disengaged and disillusioned. I understand why they have given up on their dreams. But I despair that we have a system of government at national and local level that drives away such good people rather than embracing them.

I am tired of hearing that people, whether young or old, don't want to get involved in improving their neighbourhoods. Why are we so defeatist about our social problems, when we are so confident of overcoming disease and pushing the boundaries of science? We have this huge, largely untapped, resource in this country: the people themselves. Some say the idea of inspiring people to get involved is a pipe dream. They say, rather patronisingly, that ordinary people can't make an extraordinary difference.

I don't agree. I have managed to overcome the hurdles to become a social entrepreneur, co-founding Teach First, which sends the best young teachers into the most challenging schools. I have seen so many brilliant teachers come into tough schools and elevate the lives of kids who are desperate to learn but defeated by a system that abandons the less fortunate.

So how can we change the odds and enable more people to have an impact? Firstly, we need to help more people get involved. In residents' associations or local charities, or even just a bunch of people chatting in the pub, people first experience the concept of social action. In groups, people can make change happen that they would not be capable of achieving on their own – as the anthropologist Margaret Mead said: "Never doubt that a small group of thoughtful, committed people can change the world."

Secondly, we need to eliminate the practical barriers. The state does not have all the answers, and is inhibiting change. We need to provide financing from a myriad of sources, training and support – in partnership with charities, social enterprises and businesses – to help neighbourhood groups solve complex problems like starting schools and tackling crime. And thirdly, we need to recognise that this is a long journey – real change doesn't happen overnight. One of the enduring lessons I have learned over the years is that large scale change starts with baby steps.

Ten years ago, when a small group of friends and I looked around, few graduates wanted to work in urban schools. Today it is one of the most popular things graduates choose to do, partly thanks to Teach First.

The government can only do so much. But at some point we have to say to people – over to you. If we empower them, challenge them, and support them, I have no doubt that together we can build the kind of society that we all want to see.

The author is co-founder of Teach First, a leading social enterprise group, and working with David Cameron on the Big Society