It's an important part of a doctor's job to give patients the worst news they're ever likely to hear. Why do so many still get it so wrong? Kate Hilpern investigates

You have cancer. It's terminal." My father, who died last month, was so distraught upon hearing these words that my mother never felt able to ask him whether he thought the consultant's bluntness was inappropriate. Nor did she ask my father whether he considered it bad practice that he was told at 9pm, immediately after which my mother was asked to go home because "visiting hours are over". Or whether my father found it tactless that, when he asked the consultant whether there was anything they could do for him, he was told irritably it was too late in the evening to discuss.

Back in 2001, when the journalist John Diamond wrote about his experience of throat cancer, he could hardly believe that so many doctors who regularly have to break bad news to patients had not yet found a way "somewhere between the mawkish and the unnecessarily brusque". There was, it turned out, a woeful lack of training, despite the fact that virtually every clinical speciality requires doctors at some stage to be the bearers of difficult news – anything from informing someone that their child has irreversible brain damage to telling someone they're going blind.

As recently as 2007, nearly half of consultants had received no formal training in this area. It matters, says Lesley Fallowfield, professor of psycho-oncology at Sussex University, because research shows that an insensitive approach increases the patient's distress, may exert a lasting impact on their ability to adapt and adjust to what they've been told, may affect their relationship with not just that doctor but all medical staff, and can lead to depression, anger and increased risk of litigation. Breaking bad news badly isn't good for doctors either, leading to feelings of failure, sorrow and guilt – and studies reveal it gets no easier as they become more experienced.

Lynne Johnson isn't convinced we should feel sympathetic to all doctors, however. "When I got to hospital to have a routine colonoscopy, I was shown to a treatment room, asked to remove my clothes and given no blanket, and the consultant walked in with three young medical students, not bothering to ask if I minded," she says. "After he'd looked at me, he simply said, 'I can see a growth on your bowel and I'm absolutely certain it's cancer. Put on your clothes and book a scan with the nurse.' Then he walked out."

The good news is that growing recognition of the problem means things are finally changing. All today's undergraduates and postgraduates are taught how to tell people what will often be the worst news they'll ever hear, and many existing consultants also get training. "The creation of the National Cancer Action Team (NCAT) led to medical professionals increasingly being taught the big no-nos, such as sending people straight home, saying, 'There's nothing we can do,' and telling someone on their own with no relative or friend," says Anne Corbett, end of life pathway facilitator at University Hospitals Coventry and Warwickshire. "We've since been extending that to people working with areas such as heart and renal failure."

But she admits it's likely to be some time before stories such as my father's and Johnson's become rare. "Only 10 people can do the course at a time because it involves so much role-play and it costs £5,000 – a big chunk out of any training budget." Even the NCAT training has long waiting lists, according to many consultants.

Fallowfield says the upshot is often a cheaper, less time-consuming tick-box attitude to educating medical staff. "It's all very well drawing up a list of things required – ensuring privacy, allowing for the patient's emotions and not bombarding them with information while they're still in shock. But the research I've been involved in shows very clearly that there's no earthly point in writing a load of guidelines, setting up an e-learning course or giving a lecture with Powerpoint presentations. It might create awareness, but it won't change behaviour. You wouldn't expect someone to master the intricacies of playing the cello or scuba diving by simply showing them a video of someone doing it badly and well."

Dr Ammar Al-Chalabi, a neurologist at King's College Hospital who regularly has to give patients a diagnosis of Motor Neurone Disease (MND), hopes the comprehensive training he's had will become the norm. "At my university, a fifth of the course was related to communications and since then I've benefited from a fantastic refresher course," he says.

Among the things he's learnt is that if someone asks a question, they're generally ready to hear the answer. Black humour ("Should I bother buying any Christmas presents this year, then, Doc?") is easy to brush off, but still needs to be respected, he says.

With studies showing that doctors frequently censor information they give to patients about outlook, this has become a critical focus of courses. One US study showed that even if patients in hospices requested survival estimates, physicians said they provided frank disclosure only 37 per cent of the time.

Dr Al-Chalabi adds that he's learnt that jargon is something to avoid, while silence is something to welcome. "It's easy to want to fill those awkward silences and to talk about the medical intricacies, especially if you're a natural talker like me, but what people want, at least initially, is clarity and simplicity and space to think," he says.

Mark Lansdown, consultant surgeon at Leeds Teaching Hospitals NHS Trust, agrees: "I had a recent example of a girl who needed to leave the department for an hour before she came back." He adds that he's learnt to be more aware of his own mood. "If you've opened a letter of complaint or a colleague is off sick, the pressure can be enormous, but you must suspend yourself entirely from daily pressures when breaking bad news."

Dr Tony Calland, a recently retired GP and chairman of the BMA's Medical Ethics Committee, says it's key for doctors to be prepared, no matter how heavy their workload. "When it comes to breaking bad news about children, the GP is often the first to know and we have to organise the next step before you tell the patient and also try to make sure they aren't on their own."

Bad news being broken to patients without a friend or relative to support them is a particular bugbear for Stuart Danskin, a cancer information nurse specialist for Macmillan Cancer Support. "Very often, clinics in which bad news is given are held on Fridays, too, so patients leave the hospital for the weekend not knowing where to turn to," he says.

But it's not just training that will solve such problems, says Mandy Barnett, honorary consultant in palliative medicine at Warwick University. "Targets mean that in some cancer clinics, consultants are expected to spend just 10 minutes breaking bad news. And you get some consultants who say, 'You can teach me what you like, but I'm not going to change.'"

When it became clear that my father was rapidly going downhill, I tapped the arm of a consultant doing the ward rounds to ask for an update. Despite the consultant being rushed off his feet and bound by patient confidentiality (my father never wanted anyone to know how ill he was), he took a pragmatic approach, taking me into the relatives' room and explaining that he felt it was important that someone in my family was informed he had days, possibly hours, left. He sat with me while I cried and answered every question I had truthfully and sensitively. It's hard to quantify exactly how his respectfulness and kindness helped, but there's no doubt in my mind that it did.