The illusion of control is a thing, but actual control is a thing as well. One possible reason to avoid self-driving cars is that there actually are safe and unsafe drivers, and fatal accidents in self-driving cars will presumably be a much flatter distribution among those drivers than the one we have now. Which means that even if they're safer overall, they could still be less safe if you're a good driver.