Optical illusion: Juan Muñoz's sculptures

Are they living or are they dead? It's hard to tell when it comes to Juan Muñoz's sculptures. All Tom Lubbock knows is that some of the pieces deserve a peaceful burial

The light slants down steeply from above, breaks on the straight ridge of the man's forehead, and casts both his eyes into deep shadow. You can look very closely into them and not tell if they're open or shut (though actually they are shut). This life-sized standing figure, modelled in dull yellow terracotta, has the body of a dwarf – big head; short, bandy legs. He's standing on a wooden table, which brings him up to normal head height. Under one arm he holds an inlaid wooden box, the kind that might contain a chess set. I should have said that, when the tableau is viewed from the front, the figure's back is turned to us.

This is Juan Muñoz's Dwarf with a Box, quite an early work, but like others of his figures it has an elusive presence. It feels a bit like one of us. It's not a marble figure on a marble plinth. It's part of our world. It stands on a real table, holds a real box, shares our space and our light. At the same time, these real-life props only emphasise that this is not a real man. It is a thing of baked clay, an inert and passive object, stood on its table like a big ornament, something that could be picked up or knocked over. And the blank look, the turned back, reiterate this withdrawn muteness.

When you hang around Muñoz's statuary, you can start to feel awkward in its company. You don't see these figures as safely inanimate artefacts. Their inanimateness becomes a kind of resistance or reproach. They're an uncomfortably marginal species. They're like auxiliary members of the human family, a strange supplementary race. We can't quite deny a link, but we don't know how to deal with it. Muñoz used dwarves several times in his work, perhaps because people can feel the same way about them. Dwarf with a Box is one of the strongest works in his retrospective at Tate Modern. Many of the others are not very good at all.

The Spanish sculptor died suddenly in 2001 at the age of 48. If he'd lived, he would now be 54, and he might well still be having a Tate retrospective, because at the time of his death he was already a major international artist and that sort of standing you don't lose so quickly. But as it is, the exhibition is posthumous. A lifetime retrospective will very likely be made in full collaboration with the artist. A posthumous one has other opportunities. It can decide to be a representative survey. It can decide to try and show the artist at their best. The Tate seems to have gone for representative. If it had gone for best, I think about half the show wouldn't be here.

Muñoz was certainly a very intelligent and inventive artist. His overall trick, of renovating figurative sculpture by concentrating on the dumb half-life of statues, on their stuckness and psychological opacity, and then staging them in tableaux and spaces in which the viewer can't help feeling implicated, so that your relationship or non-relationship with these figures is where the action is – that was an excellent idea.

And his subordinate ideas could be excellent, too. Muñoz picked up on the form of the "weeble", the child's toy with the heavy rounded base that you can't push over, and used it as an inspiration for his sculpture. Great notion, yes – the figure with the built-in plinth, that can never stand firm yet never be felled. Still, none of his actual weeble pieces are particularly effective. The work, altogether, is disconcertingly hit and miss.

You have a piece like The Wasteland: it's an extremely economical and dramatic composition of statue and space. At the far side of a room as you enter, there's a little bronze man, like a tiny midget or a ventriloquist's dummy, sitting tight on a little iron shelf sticking out of the wall, too high for his dangling feet to touch the ground.

But between you and him, the lino floor is all a trompe l'oeil image of interlocking cubes. You're inclined to walk across this disorientating surface rather uncertainly and gingerly. In the process you yourself become an involuntary performer in the work. And then you don't know whether to see the contained little man, with his feet well off the floor, as the prisoner of this room, or its master. The trompe l'oeil could be a spell he has cast on the space around him – cross it if you dare!

Odd, to move from a charismatic piece like that, to the numerous drawings, white on black, of claustrophobic furnished rooms: they just aren't good enough to display. Or the glass cabinets full of vaguely resonant bits and pieces, which could be anyone's work. Or the works that are stymied by the presence of a recurring symbolic drum.

It's quite easy to pass these by, though. It's more irksome when Muñoz's big good idea, the half-life of statues, is turned into something silly. Sometimes his figures are powerfully mute, isolated and resistant, contained in their statueness. Sara with Billiard Table is another finely conceived composition with a captivatingly still and withdrawn dwarf protagonist. (Like Dwarf with a Box, it's in an appendix to the main show.)

On the other hand, sometimes the figures come out to meet you, show off, act up. And then, as a friend of mine said, the effect is like an inverted version of those "living statues" you meet in tourist spots – people standing stock still, under white or grey or bronze paint, striking a pose, and you're meant to go, 'Oo-er, is it a statue? Is it a person? I swear he moved just then. Oh, he's having his lunch break,' etc.

Obviously, with Muñoz's statues, the illusionism is going in the other direction, but that doesn't make any difference, you still get the same is-it-or-isn't-it tease. This comes to a crunch with the biggest installation in the show. Many Times is a room full of a hundred slightly under life-size figures, standing around in groups, in chatting mode. Each one of them has the same head – grinning, bald, Chinese. But the bodies they're attached to hold such varied postures and gesticulations that you can't quite believe the heads are all identical. Also, none of the figures have feet: they stand on the floor cut off at their trouser bottoms.

That's quite interesting, but the main thing that happens when you go into the room is immediately you feel that all around you there are people (silently) chatting. Of course, when you actually turn directly to any of the figures, the illusion is dispelled – but it still seems to be going on out of the corner of your eye. That's the point. Peripheral vision doesn't distinguish. The effect wouldn't work with only a small group in front of you, but with a whole mass of them surrounding you, you can't escape the sensation of being in the middle of a real buzzing gathering.

And it's fun, I guess, like any illusion; fun for a bit. But if it wasn't being exhibited in this august museum, but in some less tasteful art gallery (and credited to a less respectable artist) then Many Times would be instantly recognised as a piece of novelty art, made chiefly to amaze and amuse. There are several other figure-pieces in a similar vein.

It's baffling when an artist wobbles so violently. It's not a question of him getting better or losing it. Strong work and dud seem to occur both early and late in his relatively brief working life. When artists are alive, you tend to make a balance sheet, with the worse work pulling down the better. When they're dead, the worse drops away. They're only remembered by their best, even if it's just a single piece. I think a posthumous retrospective should go with this process, and edit hard, so as to put the best face on things. It may seem unfair to the artist, and the range of their output. But it's not unfair on the audience, or on the work itself.

Juan Muñoz: A Retrospective, Tate Modern, London SE1 (020-7887 8888), to 27 April