Hot metal: Anthony Caro's sculpture is showing a wonderful late flowering of creativity and spirituality

The Irish poet WB Yeats asked, "Why should not old men be mad?" as he frolicked through his old age, writing some of the most wild and exuberant poems of his life. Is the sculptor Anthony Caro, who celebrated his 86th birthday on 8 March, such another? I kept on asking myself this as I walked around a new show of his sculptures in the West End of London.

There are 13 of them, on two floors of a gallery, each one a massy, ponderous, upright thing of gently rusting steel with various admixtures of this and that – iron, copper, wood, brass. Many of them consist of great gobbets and wrenched, cloth-like lappings of steel – bits of old machine parts, steel girders, old sinks, drums, buckets, cog wheels, all welded together. Most have ended up as abstract forms. A few, the more humorous, incline towards a kind of figurative anthropomorphism somewhat akin to what Picasso used to do with baskets and old prams.

Look at a sculpture on the fourth floor called Goddess. The looming piece, with its tilting head, looks like a shamanistic nightmare, with its weird eye and its braided hair fashioned from spirallings of steel.

The catalogue calls these "Upright Sculptures" and the title of almost every one of them begins with the word "up". And so it is. We look them up and down. They are finely, aspirationally upstanding. Upwardly soaring. Upwardly mobile. They stand, challengingly, to attention, brazenly facing us down. They rise and they rise, monumentally self-sufficient. They feel on a pleasingly human scale, even though quite a few are taller than the average human. Lets call them heroically human, then. Greek-godishly human.

This has not always been the case with Caro. Many of his sculptures have hung off surfaces, or been low-lying or even sideways-slithery in feeling. They were often particularly low-lying in the 1960s, the decade in which he burst on to the scene, because they were sculptures sans plinth. That was one of the reasons why they were so revolutionary. Caro dragged sculptures down from the plinth and on to the floor. Each of the new works weighs at least two tons. The stuff of heroic hernias.

How the devil did they get up here? The gallerist points to the skylight. What is more, these are by no means the only sculptures that Caro has been making over the past couple of years. They represent barely a third of such works. The rest will shortly be shipped to galleries in New York and Paris. What could have caused such a near manic release of the creative juices?

There are two explanations – mine and that of Caro's dealer, David Juda. Mine runs as follows. For the best part of the past decade Caro has been working on a commission to adorn a chapel in Normandy with a series of large-scale sculptures linked to devotional themes. The project is called Chapel of Light and the sculptures adorn the 13th-century church of Saint-Jean-Baptiste in Bourbourg. He has made towers which brood over a high altar. He has made pieces which squeeze into niches; he has tackled a creation theme. It was an enormous undertaking. Then, suddenly, it was all done and he was free to be himself again. So there is, within this new body of work, that tremendous shout of freedom emitted by every artist when he feels that he can please himself again after having been long in service to a patron with a measuring rod and a scrutinising eye.

Caro also took away with him certain lessons which seem to have informed the spirit of this new body of work. One is to do with architecture and the idea of responding to the regular shapes of particular architectural spaces. These new pieces feel constrained by the idea of phantom upright spaces. The sculptures are fighting against them but they are also thriving within the discipline of these imaginary constraints. And within this lesson in architectural constraint, they have managed to achieve a kind of fine balance. With each piece, the various elements are held in delicate suspense, almost as if they have, at their heart, an impulse towards an enduring lightness. Even when slightly off kilter, they feel delicately poised. They seem to possess the quick rightness of good drawing. It is as if Caro has refined and cleansed his own language of abstraction. One particularly fine example of this is Drum Up which, though as ponderous and massive as ever, looks like a piece of air-borne, teetering, feather-light jugglery. That, in a nutshell, is my ponderous theory.

And what of Juda, who is standing at my elbow, looking too?

"He's bought a fork-life truck," he replies.

Ah yes, the freedom made possible by the zip and the turn of the fork-lift truck. Being in possession of a fork-lift truck means that you can move a sculpture around with amazing ease. Comparatively speaking. It means that you can put one sculpture beside another and then change your mind and replace it with something else. It means you can make – and part-make – several pieces simultaneously and shift them about and see how they look in relation to each other. Something of that kind has evidently been happening here. There are conversations going on between these sculptures. They look at ease in each other's company. They all feel as if they belong to a great, late creative surge of making. In the last 10 years, Caro has made quasi-figurative sculptures which have looked too literal-minded by half. Those problems seem to have been swiped away like some pesky fly.

He's gunning his motor.

Anthony Caro: Upright Sculptures, Annely Juda, London W1 (020 7629 7578; Annelyjudafineart.com) to 2 July. A new, five-volume overview of the sculptural career of Anthony Caro is published by Lund Humphries (Lundhumphies.com)