I see Battersea Power Station every day, and, even in its dilapidated state, it never fails to raise the spirits.

In a novel by Barbara Pym, it is singled out as being one of those grand buildings which, even in drab parts of London, provide an inspiring vista. Down the sight lines of the Queenstown Road, its four white chimneys look, on a sunny day, like a glimpse of an Art Deco future, like a Twenties lino cut of the ideal society. From the river, the coal-loading cranes are still intact, and look magnificent in their rust.

Of course, when you get up close to it, the building's decay is obvious. Giles Gilbert Scott's edifice was originally an asymmetrical, two-chimney design, operational from the 1930s; a second half, completing the symmetry with a further two chimneys, became operational in the mid-1950s. The building was decommissioned in 1983, which means that the second half of the station has now stood empty for longer than it generated electricity.

The hopes for redevelopment have gone through a number of stages, and we are just now returning to an early stage in the cycle. A Malaysian consortium of property developers has had its £400m bid for the site accepted, ahead of an offer by Chelsea Football Club. The football club had hoped to build a new stadium there, incorporating the four chimneys. The Malaysian bid is going to preserve the previous plan to build 3,700 new homes on the site and a new Underground station, an extension of the Northern Line. Chelsea are going to have to go elsewhere, and persuade a property developer in Earl's Court not to build thousands of homes there instead.

Nobody can be confident that the Malaysian offer will come to fruition. It is the fourth offer from a private company since the station was closed in 1983. An initial consortium including Alton Towers planned to open an industrial theme park, then a mix of offices, shops and homes.

The roof was removed and the project failed. A Hong Kong business then proposed a bitterly opposed development of shops, restaurants, nightclubs and cinemas. The third owner was a team of Irish property developers, who proposed a development of homes and retail; the developers went bust in 2011, and their plan, having been given approval, is now going to be taken up by the Malaysian developers.

This spectacular site could certainly benefit from some life, and the area would certainly benefit from 3,700 new homes. It's sometimes said that London is blighted by rich foreign investors, who buy property there without having any intention of using it as a home, just as an investment. Actually, that doesn't seem to be the case to any great extent. Research carried out by the charity Empty Homes in 2011 showed that London was the least encumbered with empty properties of any English region, and Wandsworth, where the station is situated, has the smallest number of vacant properties of any English borough, at 1.18 per cent.

Those homes would be lived in, and are much more needed than another professional football stadium, of which London currently has 15. That is, if the Malaysian plan ever comes to fruition. There is a pessimistic case that the problems of the Battersea redevelopment have become so big that no private enterprise will ever be able to shoulder them – certainly, there is not much optimism surrounding a fourth private enterprise, where three previous ones have failed. The cost of developing the project has risen to £4bn; at every previous step, costs have risen so precipitously that it seems unlikely to stop there.

At some point, in most countries, the government would accept defeat and step in with public funds. In this country, that won't happen, thanks to ideology about not intervening, and also because public money gets spent nowadays only on large projects – with a single qualification.

It goes against the grain to allow such a large piece of London development to go ahead without some kind of sporting event attached. God knows, the East End needed money spent on it. It seemed all but impossible to find the money until we were awarded the Olympic Games, and then it flooded out. Some investigations place the costs of the spending on the Olympics, transport upgrading included, at £24bn. Would it have been better to have spent that £24bn on transport and housing anyway, without requiring the justification of an Olympics?

The solution to the Battersea Power Station problem might have seemed obvious, five or six years ago. If someone had thought to nominate it as the site, for two weeks, of handball matches and Greco-Roman wrestling, the money for its refurbishment would easily have been found from public funds. As it is, the Malaysians are going to struggle, get a little bit further, then throw in the towel. Just like all their predecessors.

A friend of mine of anarchist tendencies is fond of saying: "Let it crumble! There aren't enough ruins around. The 18th century loved ruins – why don't we have a great big one in the middle of London?"

There is something grand and beautiful about the rusting hulk, and if someone just landscaped a park around it and let it collapse, we would come to love its decay. I don't believe we would love 3,700 flats or, God forbid, a football stadium any more than that. But we're not going to be given the choice. A ruin by the river is, it seems, what we are stuck with.