It's been a dreadful week for free speech. A meeting at a prestigious London college had to be abandoned on Monday evening when members of the audience were filmed and threatened by an Islamic extremist. Then the president of a student society at another London college was forced to resign after a Muslim organisation called for a ban on a joky image of the Prophet Mohammed. Finally, on Friday, the author Sir Salman Rushdie cancelled an appearance at India's largest literary festival, saying he feared an assassination attempt after protests by Muslim clerics.

Almost as sinister as this series of events has been the reaction to them. The first has received very little public attention, despite the fact that students who belong to the college's Atheism, Secularism and Humanism Society were unable to go ahead with a perfectly legal discussion of sharia law. They'd come to Queen Mary, University of London to hear Anne Marie Waters speak on behalf of the One Law For All campaign, when an angry young man entered the lecture theatre. He stood at the front and used his mobile phone to film the audience, claiming he knew where they lived and would track them down if a single negative word was said about the Prophet. The organisers informed the police and the meeting cancelled.

The fact that in a democratic country a religious extremist is able to frighten anyone into calling off a meeting is shocking – and so is the lack of a public outcry about this egregious example of intimidation and censorship. Tellingly, what has grabbed media attention is the second incident, when a secularist organisation at University College, London came under attack for publishing an image on its Facebook page of "Jesus and Mo" having a drink together. The Muslim group that wants to ban the image got a sympathetic hearing in the media, despite arguing openly for censorship. Extremist websites, meanwhile, reacted with the fanatical language that so often appears on such sites: "May Allah destroy these creatures worse than dogs," wrote one blogger.

No doubt that kind of inflammatory sentiment was in Rushdie's mind when he decided not to appear at the Jaipur Literary Festival. In a statement read out there, the author of The Satanic Verses said he'd been warned that paid assassins from the Mumbai underworld might be on their way to the event in order to "eliminate" him. While he expressed doubts about the accuracy of the warnings, Rushdie said it would be irresponsible of him to appear in such circumstances.

Why hasn't there been a furore about all these incidents? Why aren't MPs and ministers insisting on the vital role of free speech? None of the people involved was threatening anybody, unlike the three Muslim extremists convicted two days ago of inciting hatred against homosexuals. It's been left to organisations such as the National Secular Society – I'm an honorary associate – to say that a fundamental human right is being eroded in the name of avoiding "offence".

Most people in the UK don't condone violence, but a worrying number think we should be careful around individuals with strong religious beliefs. This argument is mistaken, because it suggests that believers aren't as capable of exercising, or under the same obligation to exercise, judgement and restraint as the rest of us.

It's also based on fear, tacitly acknowledging a link between demands for censorship and threats of violence. One often leads to the other, and it isn't just atheists and secularists who should be very worried indeed about that.