The British summer has finally arrived: the temperature has reached 29 degrees; winter coats are on the rails at M&S, and the schools have closed for the long holidays. The universities were vacated weeks ago.

How empty an out-of-term university can be I discovered again recently, wandering the echoing corridors of what used to be Queen Mary College and now styles itself Queen Mary, University of London. The lights were on; the lifts worked; there was a porter who told me where to go. But it was as though he was amazed to be stirred. A few students, or so they seemed, sat outside a café that wasn't working.

My destination was a room in just one of many buildings, old, new, and under construction, that crowd on to this site in east London, not a million miles from the Olympic Park. Outside the perimeter, the streets thronged with people purposefully going about their business; inside all was calm – eerily so around the old Jewish cemetery in the centre of the campus that QM wants to redevelop. Staff and students, it transpired, were either coming in just for exams or had already dispersed for the long vacation.

It hardly needs to be said that they can't take their campus with them. But to describe the halls of residence, academic blocks, lecture theatres and sports halls as under-occupied is to flatter them. I have no doubt that, come mid-September, the campus will be teeming again, and managers will be planning more fund-raising to pay for yet more buildings to feature in even glossier brochures and house yet more students – all justified by the twin diktats of expansion and overcrowding.

It is not fair, of course, to single out Queen Mary University for riding this particular carousel of property development. A huge number of educational institutions are engaged in essentially the same enterprise: buying up adjacent sites, commissioning architects, importing cranes, and build-build-building.

A venerable red-brick university I visited earlier this year resembled London Docklands in its earliest phase of construction. Everyone apologised for the mess. But, they explained, universities had to convince future students (and their parents) that they were getting something in return for the new fees. Yet these buildings, I wanted to scream, will be empty, or at best only partly used, for about a third of every year.

While the downtime is less, something similar applies to schools, whose ambitious construction programmes were only curtailed when the then new Education Secretary, Michael Gove, found out how much money was already committed. Yes, there are schools whose buildings are badly substandard, and some 400 children in London still have no primary-school place for next term. But the schools I pass every day are currently idle and locked to the world – like their counterparts elsewhere in the country.

Oxbridge colleges, along with the more exclusive public schools, have courted the conference and summer school business with some success. The architecture and atmospherics are their own selling points. Some well-placed city halls of residence have acquired a second life as cheap hotels during the vacations – good for them. And some public schools (which have charitable status, remember) deign to allow locals to use their gyms, playing fields or swimming pools at times during the summer. But this is granted as a favour, not a right.

The mismatch remains striking. While the privileged see the vacation income rolling in, the rest cut their losses with a seasonal shutdown, even as young people complain they have nothing to do, parents cox and box work because of inadequate childcare, and primary schools prepared to bargain up the price of Portakabins.

There are solutions: schools and colleges that double as community facilities; a requirement that favoured schools share their pools and grounds as a precondition for their tax breaks; a five-term or rolling academic year, and a stipulation that no one gets permission for a new building without explaining how it will earn its keep for more than 36 weeks a year.

So next time a vice-chancellor comes soliciting funds for a fancy new building or a teacher complains of overcrowded classrooms, ask how many days a year their existing buildings are empty – and keep your wallet shut until they offer a convincing answer.