The profession is the most developed form of producer power. Because the public is often dependent upon their services, governments have wobbled between efforts to tame them and craven capitulation to their demands. Throughout the Eighties and Nineties there have been skirmishes, occasional full-scale battles, mainly under Margaret Thatcher, and frequent retreats, particularly under John Major. As the old blue-collar trade unions see their power decline, are the professional bodies adopting their traditional role?
What, then, is a profession? For all the rhetoric of the Hippocratic oath, professionalism is not about competence, codes of ethics or public service. If that were all, then professionalisation would be wholly welcome. Rather, a profession is an occupational group that secures its monopoly on its members' qualifications. That monopoly may, but need not, be backed up in law. But a profession has sufficient independence, collectively and as individual practitioners, to control its work.
While doctors, lawyers and engineers are professionals par excellence, teachers, academics and social workers are insecure in their professional status. The divided teachers have never achieved great control over education. Social workers have never succeeded in making their qualifications a prerequisite for practice, partly because their knowledge base is not codifiable or visible. The welfare state is dependent on professions that include doctors, nurses, social workers, teachers and planners, in ways that it is not dependent upon engineers, architects, and lawyers.
Margaret Thatcher used various strategies during the Eighties to curb the power of the welfare professions. Sometimes, open confrontations were staged, as when in 1989 Kenneth Clarke took on the doctors over the NHS reforms, prompting the British Medical Association to run a poster campaign with the slogan "What do you call a man who won't take the doctor's advice? Answer: Kenneth Clarke", a tactic which ended when the BMA's leaders changed tack to more low-key tactics - and accepted the main thrust of the reforms.
Sometimes professions have been pitted against each other. Family doctors and hospital consultants were set on opposite sides of the purchasing and providing relationship in the NHS internal market, and nurses were built up against both of them. The Government granted funds for a programme of university-based professional education, handed over many menial ward tasks to care assistants, gave nurses ward management tasks and access to the lay management career track within hospitals. Despite this, there is growing militancy among nurses about pay.
In other cases, the Government's strategy seemed to be to wear professions down with a series of low-profile measures. Doctors were subjected to lay managers in 1983, new contracts several times, and markets from 1991. Social workers were publicly denigrated. Although lawyers kept most of their privileges through the Thatcher years, there were small changes. Solicitors lost their monopoly on conveyancing, and won from barristers some rights to speak in the lower courts.
Tories in the Nineties lacked the energy and the appetite to confront groups of vested interests made up of influential middle-class voters. Virginia Bottomley granted the GPs concessions on their contracts and was less aggressive than Kenneth Clarke had been about the internal market. Gillian Shephard invited the teachers' unions for talks at the Department for Education. Interestingly, however, Lord Mackay has been prepared to tackle the power of lawyers in his legal aid and court reforms.
As Mr Major's government has shown itself lacking in the resolve of the third Thatcher administration, many professions are, in response, showing a new boldness. GPs recently threatened industrial action over night visits - an issue that most patients see as one of producer self-interest. Hospital consultants have forced confrontations with lay managers in many NHS trusts, challenging their right to manage. Successive BMA leaders have denounced NHS reforms, lay management, and even the heightened consumer expectations "irresponsibly" raised by the Patients' Charter.
Lawyers, too, are being equally candid about their self-interest. Some solicitors have called for the closure of the Solicitors' Complaints Bureau, one of the few recourses that consumers have against their lawyers. The new Law Society president, Martin Mears, recently argued that higher fees for lawyers are in the public interest. In the current debate about how civil court cases should be financed, the Government proposes that solicitors should be able to work on a "no win, no fee" basis, but where they will take greater fees if they win or settle out of court with substantial damages. They have demanded and won the right to charge 100 per cent more in fees where they take such speculative risks.
But does this new-found audacity by some professions mean that they are more powerful? In the Nineties, many professionals' individual autonomy has been eroded but collective self-regulation has been strengthened. As the veteran health politics commentator Rudolf Klein points out in a new book, individual doctors are subject to new controls on what they prescribe, but the BMA's grip on complaints and discipline is as strong as ever, and it issues ever more codes and guidance notes. Meanwhile, the NHS programme of quality control by medical audit, for all its benefits, has given doctors greater control over each other's work.
The autonomy of lawyers to give advice and to manage cases as they see fit has only been affected at the margin by the operations of the Legal Aid Board, the Solicitors' Complaints Bureau and the actions of the courts. Paradoxically, losing responsibility for administering legal aid to the board has freed the Law Society to concentrate much more on campaigning for the producer interest, and this has enabled it to exercise much greater control over its members.
Lord Mackay's changes to increase solicitors' rights to address the courts and become QCs and judges may slowly undermine the lawyers, but are just as likely to pave the way for a single profession that could be as difficult to deal with as the present divided one.
Those groups that are less secure in their professional status find it difficult to know how far to rely on raw trade union power and how far to rely on insider lobbying as the doctors and lawyers traditionally did. The temptation to exploit the weakness of the Major administration by behaving like a trade union can be very great. Teachers, for example, after several years of moderate leadership, have focused on the issue of class sizes in the hope of taking back from government the position of defining what is good and bad in schooling.
Disputes this year with nurses and midwives over pay have shown a new boldness by these professions in exercising industrial power.
More and more middle-class people like to think of themselves as professionals. New degrees appear every year for occupational groups aspiring to become professions. Various management groups are becoming professionalised. Personnel management has long had a professional institute and a qualification. Financial managers, information technologists and managers, auditors and management consultants are gradually developing accreditation, autonomy and organisation and public sector influence that is making the state increasingly dependent upon them. Governments will soon have to treat them as professions.
So what might be the next government's strategy? If there were to be a new Tory government, opinion polls indicate that it would have a reduced majority. In that case, it would be most unlikely to return to Thatcherite confrontation. Right-wing populists, such as John Redwood and the 92 group, are concerned to secure middle-class support and will shy away from anything that would alienate "heartland" votes. Labour, traditionally the party of the welfare professionals, has moved only very cautiously under Mr Blair to challenge professional power.
What could be done? Certainly, more use of internal markets in health and contracting in social services, lay managers to carry out purchasing in health and legal aid, and the assertion of state power to define quality are part of any programme. But the key elements of a strategy to tackle the welfare professions must be a challenge to their autonomy. As long as government is committed to paying for whatever doctors choose to prescribe NHS patients or lawyers choose to advise people on legal aid, the public finances can be held to ransom. No similar concessions of autonomy and control must be made to the new professions. Already, the power of auditors to require policy changes in local government and other areas of public service is very wide. Above all, for every service at risk of being professionalised, government must create both competition and independent powerful consumer agencies.
Governments of the next decade will be squeezed between fiscal and consumer pressures to face down the professionals and the power of the professions to persuade people that they understand the public interest best. The challenge for the parties is to commit themselves to long-term strategies for controlling these powerful interest groups.
Perri 6 is Research Director at the independent think-tank, Demos.