A passenger is aware of the road conditions that the driver is dealing with. In potentially hazardous situations he or she will usually have the sense to shut up. A child can be told that Mum or Dad is too busy to talk just now.
Telephone callers are not aware of the traffic conditions, and it is not so easy for the driver to request them to shut up, particularly if they are senior colleagues or customers. That is why hands-off phones, though preferable to the hand-held variety, are still a distraction that should not be used while driving.