Why We Build, By Rowan Moore

If you build it they will come, but they might be a bit of a shock

Sir John Soane's Museum in Lincoln's Inn Fields in London is, in the opinion of its architect, a failure. Soane had intended the three conjoined houses not only to be tribute to his professional skills but also a family house which would inspire his sons, John and George, to follow in his path. Neither did; the first died young and the second wrote anonymous attacks on his father's abilities as an architect. In an act of spite, Soane left his house and his collection of antiquities to the nation, and died unfulfilled.

He was not alone in discovering that, however well designed and solidly founded, a piece of architecture rarely achieves what is intended for it. Rowan Moore's ambitious new book Why We Build should really have a question mark at the end. From Dubai, a city built on sand and speculation, to the author's own aborted attempt to have Zaha Hadid design the new building for the Architecture Foundation, of which he was director, it is a catalogue of thwarted human desires. Had the outcomes been known, many an architect and patron would have shredded the plans.

The Earl of Bedford, for example, commissioned Inigo Jones to redesign Covent Garden in the early 18th century. It was intended to be a piazza for the well-to-do, whose rents would line Bedford's pockets, but a raucous market expanded into the vast new square, gambling dens sprung up, and its arcades and colonnades proved perfect locations for the prostitutes of Fleet Street to solicit from. Covent Garden became a playground for the baser human pleasures.

One Hyde Park, the Candy brothers' eye-wateringly expensive modern development on Knightsbridge, took the opposite direction. The lozenge shaped buildings designed by Richard Rogers are so austere and have such high security that the life has been stamped out of them.

What drives architecture is very much what drives our normal existence: the hope for a happy family; a desire for wealth; displays of vanity. But successful architecture is often achieved by accident. The 1960s Bijlmer estate south-east of Amsterdam or the stucco fronted Victorian houses in London's Notting Hill – both heavily planned residential areas – were abandoned by their intended inhabitants and left to the poor and new immigrants. Both were reborn only after a new wave of occupiers, far from the desired ones, breathed life into them.

Moore does hold up some guiding lights. One is the work of the late Italian-born Brazilian architect Lina Bo Bardi, the designer of the São Paulo Museum of Art, in which he sees a sensitivity to the idea that buildings are meant to be adaptable and pervious to human or natural intrusion. Another is the High Line garden walkways built on disused railway lines on the west side of Manhattan, that make a virtue of the structure's own redundancy .

As any book attempting to write about the nature of human desires, Why We Build is complicated and contradictory. There is no blueprint offered. As John Lennon put it, life is what happens while you're busy making other plans. What does emerge is a subtle argument to the effect that those who look to architecture as a means of mastery set themselves up to fail, but those who leave the outcome open may be pleasantly surprised.