Sir Norman said the admission by Mr Yeo that he fathered a child outside marriage did not affect his position as a minister who has been tipped to join the Cabinet.
He said: 'I do not believe it is part of our role, and I don't think it is part of the media's role, either, to go into the private lives of individuals, unless it has a dramatic effect on the role the minister is carrying out.'
There have been calls for the minister's sacking from Tory and Labour MPs, but Downing Street has made it clear that John Major, the Prime Minister, is prepared to support Mr Yeo over his affair with Julia Stent.
Ms Stent, 34, a Conservative councillor in Hackney, east London, gave birth to a daughter in July.
Mr Yeo's admission comes in the wake of reports that Steve Norris, the minister responsible for transport in London, was engaged in a series of affairs. The latest defence of Mr Yeo indicates that, unless a major scandal erupts, it is unlikely that a minister will be expected to resign in such circumstances.
Mr Yeo once told his local branch of Relate, the national marriage guidance organisation, that it was in everyone's interest to reduce the number of broken families and single parents.
But Sir Norman denied in a BBC radio interview yesterday that revelations about extramarital relationships damaged the Tory party.
'What government is about, what the Conservative Party is about, is developing public policy, which has an impact on the public generally.
'It would be a great mistake if we went down the course of trying to pillory people for their own private lives,' he said.