When I was 14, I passed through the main cafeteria section of a youth club I belonged to. A group of rather good-looking teenagers were sharing soft drinks. As I left the room, I heard the words, delivered sotto voce: "There goes Franco." All the teenagers laughed.

It took me a few seconds to work out what the sentence meant. Having dismissed the possibility that I was being compared with the infamous Spanish dictator, I realised that Franco was a diminutive of "Frankenstein". I surmised that this was connected with the fact that I had scars on my lip and left cheek.

Did that moment affect the way I saw myself for the rest of my life? I don't know. But I do know that I am writing about it 40 years later, as I remember it so vividly. Just as I remember, when I first got married, one of my friends describing me as the "runt of the litter" (my two brothers are mercifully better endowed in the looks department).

Samantha Brick caused a storm in the Daily Mail when she suggested that women who were not beautiful hated her for being beautiful. Men, she seemed to conclude, did not because they were less vain: they did not compete. They were above such things.

I think it is an illusion that men are immune from concerns about the their own physical appeal. But this concern about personal appearance happens to be taboo among men. A friend confessed to me once that he spent a vast amount of time worrying about his appearance. It was something he had never admitted before as he was embarrassed by it. He would be accused of being gay, vain, shallow, self-obsessed, and who know what else, if he ever "came out".

Women, however, can be openly obsessed about appearance and be seen as entirely normal. So I am coming out of the closet on behalf of all my male brethren. Do I care how I look? Absolutely. I weigh myself every week, and become duly depressed at the gradual broadening and softening of my body. I look in the mirror and feel my mood slump when I see the bullet-headed, lopsided, wrinkled and misshapen visage that gazes back at me – just as I always have.

So what am I? Vain, insecure, effeminate? And what am I worried about, exactly? This is a question that is also usually asked only of women. Do they try and look good for the sake of other women? For the sake of men? Or purely for themselves? And which is more "morally defensible" – since it seems that you need to defend yourself morally for being unduly concerned with appearance?

At the age of 56, I am a codger, sexually invisible. And yet, I continue to worry about how I look. Every day, the mirror casts its accusations. Every week, the scales hurl their reproaches. Feeling as I do, I think I have some insight into what the majority of women feel about the subject (and perhaps a surprisingly large number of men). After all, what quality is more effortless, more powerful, more ubiquitously potent than beauty? Children survive childhood because they are beautiful – at least in the eyes of their parents and for the most part in the eyes of other adults.

Meanwhile, adult beauty, in our primitive reptilian brains, speaks of reproduction, of good genes, of vitality and fertility. No wonder research shows that attractive people do better at work, have more friends and report higher levels of happiness than the rest of us. They don't have to do anything. Like children, beautiful people just have to be.

This is perhaps why beauty produces such violent reactions. Instinctually, we are drawn to attractive people, as a kind of status symbol. At the same time, however, both men and women respond to beauty with a form of revulsion. What did the beautiful do to earn this magnetism, after all? Being beautiful is just so unfair, it irks all the rest of us who do not belong to the 5 per cent or who make the grade (and it feels a far higher percentage because of the power of media and advertising).

You can be born poor and make yourself rich, you can be born vulgar and become sophisticated, you can start out obscure and make yourself famous, you can be a failure that turns into a success. But nothing, really, is going to make you beautiful. With dieting, make up and clothes, you may produce a workable simulacrum – but it's not the same thing.

We all tell our children that looks don't matter. In fact, I have just told my nine- year-old that very thing. She responded curtly. "Of course they do." Like fairies, and Father Christmas, she knew this was another myth that adults peddle to make the world feel a fairer, better place. The importance of what you look like is inescapable, and your level of attractiveness is almost invariably reflected in your partner.

The hatred of unattractive people is one of the last sanctioned forms of abuse. From films such as The Hottie and the Nottie and The Truth About Cats and Dogs to the websites that post your photo so people can vote on whether you are a "minger" or "fit", the aesthetically challenged are fair game. In the schoolyard you can call someone "butters" without fear of reproach.

I do not deny that, in the long run, personality counts for more than your looks. If it did not, I would not have managed to couple with a reasonable number of good-looking women. But being intelligent, interesting, nice, writing books and so on is all such damn hard work. Whereas if I looked like George Clooney, everyone would want to be with me, even if I was a lazy, cross and smelly.

So what if women hate Samantha Brick for being beautiful? Everyone gets hated for something. I am usually hated for being too smart with my tongue, or too nosy, or too tactless. All virtues are double-edged.

I truly wish, however, that beauty was the fault for which I was called to account in my life. The wonderful thing is that it gets you noticed – and that is the secret hope of everybody, as Arthur Miller put it, to "transcend anonymity". You are visible, you exist as a body, even if your mind is as empty as a gourd.

As I get older, my contemporaries have come to look as bad as I do, or worse. We are all invisible now. But it is better to have been fit and lost your fitness than have never to have been fit at all. My mother always reassured me when, as a child, I asked her if I was good looking, that I had a "good personality". I wish she had just lied and told me I was gorgeous.

I would have probably believed her, just as Ms Brick obviously believed her parents. Because if there's one thing better than being beautiful, it's believing that you are.

Tim Lott's novel, 'Under The Same Stars', is out now (Simon and Schuster)