Tim Lott: Ideology buckles – it's too rigid, too flawed. Better by far to muddle on

After the last traumatic weeks spent watching the near-fatal buffeting of the world economic system, it was fascinating to read of the commensurate bruising endured by the mind of one of the principal architects of that system, the neo-liberal former head of the Federal Reserve, Alan Greenspan.

Greenspan admitted to a congressional committee last week – after the collapse of the 40 years of economic experiment in the free market economics that he had done much to establish – that he was "shocked" at the failure of that model. He was "very distressed"; it was a "tsunami" that was "much broader than anything that could have been imagined". Most tellingly, he admitted that what he had believed in was an "ideology" which, he now recognised, contained a "flaw".

Related articles

• Victims of recession: Surviving the credit crunch
• Bloodbath on Broadway: The credit crunch hits
• David Strahan: You're wrong, PM. We need higher oil prices
• So you're bullish that shares will rise up off the canvas?
• Julian Knight: Don't give control over rates back to the politicians
• They've raised the bar if you're after a loan
• Obama and Brown urged to join hands on financial regulation
• Europe's interest rates to be cut again as recession bites
• Margareta Pagano: The Bank can do its job. What about the rest of them?
• Kenneth Clarke: The once and future Chancellor?
• Hamish McRae: Recessions hurt us all, but how unfair if Africa is punished for Western excess
• Search the news archive for more stories

It is an extraordinary thing that such an intelligent man should be so shocked that an ideology contained a flaw. The shock came because, presumably, until the last few weeks, Greenspan didn't see what he believed in as being an ideology at all – that is to say, a self-contained, self-sustaining, supposedly coherent and hermetic system of belief. He saw what he thought constituted reality. But what he actually saw were patterns made out of smoke.

And that is why he is now reeling, along with the economies of the world and all of us that rely on them to keep our daily lives tractable. Because reality has revealed itself as too unpredictable and complex to contain within a single rigid body of ideas.

No ideologies really admit to being ideologies until they have failed. It wasn't only Greenspan but a whole generation of libertarian economic theorists who were gripped by a vision arising out of the failures of the centrist, statist "managed economies" of the 1970s.

The emergence of this mixed economy statism – too patchwork to be described as an ideology – was prophesied in a book called The End of Ideology by Daniel Bell in 1960. He observed that both the ideals of the left, in the form of socialism, and of the right, in the forms of fascism and militarism, had expired, leaving only the possibility of a raft of centrist, moderate and ad hoc compromise politics negotiated between competing centres of power.

This model – which came to be known in this country as corporatism – was seen to have failed after the inflation, economic stagnation and oil crises of the 1970s. Once again, a vacuum of ideas, or if you prefer, ideals, was waiting to be filled. And filled it was, this time by the libertarians, among them one Alan Greenspan.

This moment of the shift to the belief in economic libertarianism is captured in an article by the libertarian guru Murray Rothbard in 1977 in a journal called New Libertarian Forum. Pointing to the failure of centrist corporatist solutions, he observed that "all this provides an unusually favourable opportunity for libertarians. For we are functioning in an intellectual climate where there is no longer any real, determined, militant ideological competition. Ideological decay and confusion are everywhere. But, in this miasma, we libertarians have that alternative; we have a new and intellectually stimulating and fascinating ideological paradigm, and one that explains the collapse of modern statism better than anyone else. We have a new and systematic creed, and we are just about the only ones who still believe in our ideology. In contrast to the Left, Right, and Centre, our ideology hasn't ended; it is just beginning."

The triumph of the libertarian view came with the fall of the Berlin Wall in 1989, after which American-style free market economics was the only game in town. Francis Fukuyama, echoing Bell, published The End of History in 1992, asserting that democracies underpinned by capitalist economies were now the only realistic way to organise a society. All other models had failed. The belief in this creed continued to be so irresistible that it was ripe for export, by warfare if necessary, and fed into the partially ideological wars in Afghanistan and Iraq.

Now the dream has ended, as dreams must. They must because ideology implies a world that can be finally and neatly explained by systemic belief. And it should have been clear that this was impossible after the failure of so many ideologies during the 20th century. But we continue in our fantasy and our hope that a grand clarity of vision might somehow be attainable.

My dictionary defines ideology as "the body of doctrine, myth, belief, etc, that guides an individual, social movement, institution, class, or large group". The use of the word myth is instructive. For ideology – and free market economics, Marxism, Fascism, and every political system that advertises itself as based on certainty and a coherent set of ideas – is, at root, myth. It is myth because the world we find ourselves in is too complex fully to understand or explain.

The trouble is that, psychologically, we seem to demand myth. Without it, we feel threatened with paralysis. Pure pragmatism, the stumble and blunder of everyday confusion, has no enduring appeal. The 1970s anti-ideological age (at least in government) ended in stagnation and failure.

Ideology provides what political scientists call a "mobilising myth": a story that will energise individuals and groups into action. For good or (as it is proving) evil, myths provide a platform for energy, for dynamism and movement. As Daniel Bell himself pointed out in a 1972 article, "there is always an emotional hunger and yearning for ideology and ... these impulses are always present among young intellectuals". This was true of libertarian Western intellectuals in the 1970s. It might also be worth pointing out that it was true of the middle class prophets of a very different ideology, most notably the Islamist ideologue Sayyid Qutb, and later, Osama bin Laden.

So where do we go now that ideology has landed us in the soup once again? Was corporatism really such a bad idea – the politics of consensus and muddle and hedge? Yes, it collapsed as competing pressure groups – notably the trade unions – pushed their claims to the limits, the oil dried up, notions of class on this side of the Atlantic stained everything, and respect for fairness took a back seat to the competing demands of organised individual groups.

But might it be that we have had the opportunity to learn our lessons now? That we could make corporatism work? I'm a great believer, despite having lived through the disasters of the 1970s, in pragmatism and muddling through with fairness and – that much sneered-at notion – common sense. The trouble is, although it is traditionally a very English message, it isn't a very sexy one.

Corporatism is a failed model – but pragmatism, modesty of ambition and a sense of fairness can never ultimately fail because they are rooted to my mind in something real and enduring: that is, human nature. Boring old pragmatism is always the lesser of evils. It's better than anything else because it works, or can work, more or less. Or, at least, perhaps it can work without causing too much damage. None of these are remotely sexy or exciting statements – "Fudge!" Margaret Thatcher would have exclaimed – but I suspect that's the best we can do.

But rather than concentrating on which system or systems might replace the failed model of laissez-faire capitalism – I'll leave that, all experience to the contrary, for the "experts" – I want simply to point out that we need, above all else, to find a way to recognise and live with uncertainty. And anyone attempting to rebuild our recently shattered paradigm of economic organisation should take this into account before sitting down to make a New Exciting Blueprint of Society for us all.

Pragmatism, caution and regulation may – in fact it will – have unfortunate consequences. But they are unlikely to lead to disaster of the kind we have faced in the last month. Pragmatism is, if you like, the lesser of evils, because it recognises unpredictability – the "black swans" that Nassim Nicholas Taleb so famously predicted – as a fundamental and inevitable characteristic of human society.

Throughout human history, the urge towards certainty and grand schemes has perennially led to tragedy. We need to find humility in the face of the formidable complexity of the world. Only armed with that humility have we got a hope of surviving the revenge of invented, hubristic systems – such as we are now experiencing. For those systems always turn out to be far less benign than we hoped when, out of fear or simple wishful thinking, we conjured them– brilliantly, but fatally – from the limitless caverns of our own minds.