Umbrella is set across an entire century and follows a misdiagnosed woman in a north London mental hospital, her family and her doctor. Judges said that the novel, which has no chapters and few paragraph breaks, was both "moving and draining".
"Those who stick with it will find it much less difficult than it first seems," they said.
Novelist and journalist Self, 50, appears on the shortlist for the first time and has been installed as favourite to win by bookies. His previous books include How The Dead Live, which was shortlisted for the Whitbread Novel of the Year, and The Butt.