Downing Street led attempts to rally round the environment minister by making it clear that John Major regarded Mr Yeo's affair with a Tory councillor as a 'private matter'.
As that message sank in with Tory backbenchers, who do not return to Westminster until 10 January, Mr Yeo's chances of holding on to office appeared to improve.
But David Evans, a newly-elected member of the executive of the 1922 Committee, which represents Tory backbench opinion, made it clear that Mr Major would be making a mistake in allowing Mr Yeo to remain in office.
''Any minister or MP who does not behave in a way that nine people out of ten expect them to behave undermines the Prime Minister every time they do it, whether it's drinking and driving, being a homosexual, making love to councillors or half a dozen reporters, knocking off part-time actresses in Earls Court, it all undermines the Prime Minister,' Mr Evans, the MP for Luton South, said.
His warning about Mr Yeo, a leading liberal in the Tory party, may signal a backbench move against Mr Yeo by the right of the party when the Commons returns.
'We are the party of family values. Anybody that sets themselves up in public life has a duty to set an example. In the sort of society we are living in, more and more we need people to set examples.'
Mr Evans pointed out that Mr Major's standing had been undermined by sex scandals in the past, including those involving David Mellor, who resigned from the Cabinet after an affair with an actress, and Steve Norris, who remained in his post as minister for transport in London after reports of a series of affairs.
'If you accept the privilege, responsibility and honour as a minister, you have standards to set and unfortunately in the last year or so several of our ministers have not set that standard. For me, (it's) 'goodbye'.
'You don't drive through red lights, you don't drink and drive if you are a minister, and you don't go knocking off everybody,' Mr Evans said on BBC radio.
Patricia Fitzpatrick, Mr Yeo's Suffolk South constituency chairman, said his constituency executive would rally round Mr Yeo, who left for a foreign holiday with his wife in order to escape the publicity.
'He has always served the constituency very well. His personal life has not affected the constituency at all. I shall be speaking to him in the next few days after the Christmas break and I shall be speaking to my officers.
'He has been more than honest with what he has said.' She said it would be 'very surprising' if the officers did not rally round. 'He is a very popular member. We have all been extremely happy with him.'