In the ancestral environment, violence was the main cause of death you could do something about and cars didn't exist. The result is evolved instincts that are wildly irrational in the modern environment.

If you actually stop and think about it for a minute, the probability of the driver taking it into his head to murder you is likely negligible compared to the probability of being killed in a plain ordinary road accident, despite - or because of - the fact that we have no instinctive fear of the latter.