Man is a physiognomist. We tell ourselves we shouldn't judge a book by its cover – though, speaking as an author, I must tell you that my covers have caused me many sleepless nights – that appearances are deceptive, that "there's no art to find the mind's construction in the face", that the best of us have "that within that passeth show", but the truth is, show wins it every time.

In the days I used to help my father on the markets, we called the way we set up our stall a "flash". A good flash meant good takings, no matter what it was we were flashing. Pre-torn willow pattern plastic Taiwanese tablecoths, chalk 3D wall plaques of the Last Supper in which the Apostles appeared to be eating chicken chow mein, nests of tartan cardboard suitcases from Romania (what happened, by the way, to the concept of a "nest" of suitcases?) – none of them irresistible objects of desire in themselves, but we had only to flash them well and out they flew.

Once a flash man, always a flash man. I remain a lover of attractive packaging, on people as on market stalls. And I am not alone. It was Helen of Troy's face that launched a thousand ships, not her inner qualities. I wrote the other week of a dinner I went to in the Long Room at Lord's but omitted mention of Alastair Cook whose height and smouldering Anglo-Welsh good looks caused normally cynical and collected women to lose their reason. "Look at him – he's a god," one of them breathed tropically in my ear. "No, he's a man," I reminded her, "with a tendency to plant his foot and a somewhat faulty technique as witness the number of times he has been caught behind outside off-stump." But did that change how she felt about him? Not one jot. Man is a physiognomist and so is woman.

Acquired long ago in the primeval soup, it's a judgemental shorthand of which we shouldn't be ashamed, for much of the time the way a thing or person looks is what a thing or person is. It took quite a few slow-motion replays to be sure that John Terry really did knee Alexis Sanchez in the back the other night in the course of Chelsea's remarkable victory over Barcelona. In fact, it was a linesman who reported the incident to the referee, but even he needed a quick eye to be dead certain that Sanchez hadn't just lost his balance, toppling back into a knee Terry had raised only to cushion his fall. So was it Terry's reputation as a brawler and all-round not-very-sweet person that caused him to be sent off? My suspicion is that it was his face.

Something in Terry's expression, perhaps in the narrowness of his eyes, perhaps in the thinness of his lips, perhaps in the determined way he sets his jaw, perhaps in the way he carries his head as though it's a weapon, perhaps in the impression he always gives on the field of having just sworn at an opponent, a spectator or a team-mate, something, let us say, in the sort of street-weary anger of his expression, as though he feels it's a dog-eat-dog world out there and if you don't get in first with your knee someone else will get in with his, simply screams red card. If I were a referee I'd bypass the niceties and red-card John Terry before a ball was kicked and let Chelsea get on with it with 10 men.

I am having comparable difficulty, to go from the sublime to the ridiculous, with Jeremy Hunt, Secretary of State for Culture, Olympics, Media and Sport. His face, too, has me reaching for a card, not on account of any innate aggression, but because it is unsuited to the political function which, as a face sitting on the shoulders of a minister, it exists to serve. It is too jaunty. It seems not to know what all the fuss is about. It is a face, however one describes it, on which none of the gravitas we have the right to associate with culture (forget the Olympics, media and sport) appears to find traction.

Whether or not, in the matter of weighing the advisability of News Corp taking over BSkyB, he will finally be shown to have acted "quasi-judicially" – the phrase with which Harriet Harman has been hounding the nation to the point of madness all week – it was evident both from what he said and the manner in which he said it on taking over from Vince Cable that he neither objected in principle to the bid nor understood why anyone thought he should. It has been argued that every politician has an attitude of some sort to News Corp, and this needn't matter so long as the proper judicial process is adhered to, but it does matter which attitude to News Corp you have and Jeremy Hunt had the wrong one.

Relativism kills with lightness, and Jeremy Hunt has a relativist's face. In this, you could say he is the polar opposite to John Terry, for whom existence seems never to be relieved with lightness. Lucky is the man on whom serious matters don't weigh heavily. But not when he's a minister. Hunt's face lets him down not only because it fails to express deep consideration but because it goes blank before the deep consideration of others. He comes, therefore, in the wrong packaging for a Culture Secretary.