The Music Room, By William Fiennes

If you put Broughton Castle into Google, you'll find it is a moated medieval gem near Oxford – with garden tours, teas served and all the usual stuff. You'll also find that it was the setting for a variety of films, from Shakespeare in Love to The Madness of King George. So reading William Fiennes's book, about life in the castle as a young boy, with his brain-damaged older brother, is rather like opening a locked door in one of its corridors and finding a secret, cobwebbed room into the past.

Fiennes's early life was near-magical. There were gardeners, helpers in the kitchens, swims in the moat (with lily-pads "sitting on the water like jam papers"), learning to bicycle in the Great Hall, scary moments in the Groined Passage. There were memories of his father sitting beside the fire reading Everyman editions of Trollope, his mother sewing patches into his old tweed jackets, with pieces of leather "the same size and shape as the panels you punched from Kleenex boxes to get at the tissues". But his life always dominated by the presence of Richard, older by 11 years.

Richard had always had epileptic fits from childhood but one full-blown attack resulted in his frontal lobes being damaged. As he grew older, he turned into an unpredictable, powerful, sometimes delightful and sometimes frightening presence. One moment he could be out watching the herons on the lake, contentedly tamping his pipe with special tobacco, and raising his Leeds mug with a shout of "I'm proud to be a Leeds supporter!"

But the whole character of the day would rest on the result from Eland Road. If things went badly, he'd be glowering and swearing if anyone so much as asked him to come in to lunch. "You and whose army?" he'd shout, if anyone crossed him. "Watch your mouth!"

His great strength and unpredictable aggression meant that he might yank the cars key from the ignition and throw them in the moat, cut the telephone wires with kitchen scissors – once he even seized an iron bar and smashed the windows. These attacks would be followed by pitiful remorse. "I'm a horrible person," he'd say. "He could swing from gentle benevolence to aggression and back again within a few hours, as if a different personality had taken him over and then given ground. But we didn't see this as anything unusual." Fiennes could not imagine it as a disease. "That would have implied the existence of an ideal healthy Richard... But there wasn't any other Richard."

Eventually, Richard was thrown out of the epilepsy centre where he had spent some time, because he was "very aggressive, just waiting to have a go at someone." Luckily he found some kind of peace before he died, tending the plants in a caring centre which accepted him. The young William was in a strange position. As he grew older, he started to realise that he'd grow out of his stage of childishness while Richard's childlikeness was indefinite. He was, like the castle, "moated in".

Apart from the unnecessary padding in which, from time to time, Fiennes writes about the history of epilepsy, this is an exceptionally honest, beautifully-written and observed memoir of a strange childhood, touching in its description of a situation about which, while others might have moaned, appears to have been simply accepted and absorbed, as best it could, into daily life. This is no misery memoir. It is a memoir full of curiosity and affection.

Virginia Ironside's novel, 'No, I don't want to join a book club!' is published by Penguin