"People care, so I have a whole wodge of letters, which I haven't had for a while," he said. "I'll reply to them all, ring them, have a chat, there are a lot of balanced arguments. People are upset because it is the denting of dreams which can be hard to take."
Graveney seemed to be bearing up pretty well last Friday but he also knew that pressure on him had increased in direct proportion to the margin of defeat. There is probably some correlation between weight of expectation and resultant weight of defeat, and 239 runs is the basis of quite a handy critical missive.
"I've taken a bit of a hit. I feel for every fan but maybe a little more realism can come into the conversation. We have got to be realistic, we are the second-ranked team in the world which is a fantastic achievement by the team and the coaching staff. We don't have masses of people high in the individual rankings so it is a testimony to the spirit of the dressing room.
"The selectors have discussed our options. We just calmly looked at what we had to do. I am hopefully not cast as an idiot but I also know that everyone talks about this magical phrase 'consistency of selection', which is great when you win. But the real test comes when you have an adverse result. That is when we have to hold our nerve and make the right decisions."
Of course, he cannot reveal the secrets of the selection room but whatever team is announced today it is a fair bet that changes were discussed. Paul Collingwood has leapt into the public domain because of his competitive edge and three big centuries for Durham this season (four as of Friday). But immediate change might indicate either panic or indecisiveness.
These are once more difficult times for Graveney. He got it in the neck - and is extremely annoyed - for Graham Thorpe's dropping and subsequent retirement. He did not want Thorpe to quit and is perplexed at his attitude. It may have been wrong given such long experience but selectors have to make tough decisions. There are two obvious points: Kevin Pietersen, Thorpe's replacement, had a wonderful batting debut and by walking away so quickly Thorpe probably proved the selectors were right.
Graveney's role is no longer as overpoweringly influential as was traditional for the chairman of the England selectors, but he remains more than an adviser. In any case, if they lose, the public expect him to get it right. The panel, which now consists of Graveney, Geoff Miller and the team coach Duncan Fletcher, met last Monday.
They talked about the game just gone, about how they might play now. Usual stuff, then. But they had to consider what it might do if they made any changes so soon. It might be bold but what if it created an environment where people are forever looking over their shoulders? On the other hand, as Graveney recognises, it could be too late by the Third Test.
This is his ninth summer as chairman of selectors, the longest period since Alec Bedser straddled the Seventies and Eighties. He has been chairman for 101 matches. Like every chairman before him, he is not universally popular, although it is probably fair to say that not every individual selection is down to him. He is disparaged unnecessarily sometimes but there is no doubt that you do not survive as he has - and he is known as a great survivor - without certain skills.
However, as much can be said for Fletcher, who invariably deflects questions about selection in Graveney's direction, not always to give credit where it is due. The relationship between Fletcher and Graveney is a source of some fascination among England watchers. Neither man is for drawing on it, but they have never been seen in a fond embrace during any of the long round of victories before Australia arrived and there may be reasons for that.
Graveney can hardly afford to write England off but he has to be clinical in his judgement of the relative merits. "I thought in the lead-up how big the gap was between one and two and we wouldn't know the answer until we put them together. In very much heart mode I said the margin would be 2-1 to us but I suppose if I'm using my brain that if the Ashes are still alive when we get to The Oval we have done exceedingly well to get to that position."
The chairman will go to Edgbaston early this week and speak to the players about the reasons for the selection. Mostly, he will remind them why they were selected and perhaps put an arm round their shoulder. "It will be interesting to see how they rock up on Tuesday. I will make the point that it has been a tough week but we've all got to be positive and back our belief, in terms of selectors but above all the players themselves. They have just got to clear their heads and trust their ability. How you cope with a nation's expectation was always going to be interesting and this is a nation's expectation."
He spoke similarly about Michael Vaughan, the third captain he has appointed, and what a huge job being captain of England is. "He's batting like a bloke with a lot of things in his head. Somehow he's got to clear it. It's tough at this level and playing this lot is another level up."
Graveney claims convincingly still to love the job - and it is his job - although it is known that he expressed an interest in the job of chief executive at Somerset. Whatever the turmoil in Taunton it was bound to have been more tranquil.
"I can understand people's frustration and disappointment but maybe we have to reduce the hype a bit. Be assured this is a team of huge character. They can bounce back. This might not be of any interest to the public, but I stopped smoking before Christmas and I haven't restarted."
If smoke comes from the direction of Edgbaston then it will be signalling something less joyous than the election of a Pope.