One reason I've heard is that when you're on the phone with someone, they don't know what's going on in the car whereas if you're talking with the person next to you and you suddenly need to focus all of your attention on the road, they can stop talking rather than adding to your distraction. Plus they provide a second set of eyes. I don't know how accurate and reliable that theory is, but it makes sense to me.
I also wonder if the mental effort of listening to a voice coming through on a speaker would make it harder to focus. I tend to have trouble hearing people when I'm talking on the phone; but that might just be me.
It's no more distracting than talk radio or some blaring annoying commercial which doesn't stop talking if I need to focus. When you have a handsfree device, just say "Hold on" and do what you need to do. You can always ask them to repeat themselves I you don't hear.
I also wonder if the mental effort of listening to a voice coming through on a speaker would make it harder to focus. I tend to have trouble hearing people when I'm talking on the phone; but that might just be me.