In the BBC Radio 4 programme, In The Psychiatrist's Chair, Lord Parkinson tells Dr Anthony Clare he did not think it would do any good to see her.
'The relationship between her mother and myself disintegrated to such an extent that I just didn't think it would contribute anything to her life,' he said.
He also said he could not explain why he began the affair with Sara Keays, his former secretary.
In his conversation with Dr Clare, Lord Parkinson said: 'I really don't know. All I know is that I did. That's a part of my life I haven't attempted to try and explain to myself.
'It's something that happens to a lot of people, especially if they have the opportunity. Politics is a very unaccountable profession in a lot of ways, in that your life is so unusual - the House of Commons late at night, one has scope.'
He said that although he promised to marry Ms Keays, he changed his mind on a family holiday.
The programme will be broadcast on 8 August.