Personally, it seems I touch wrong buttons quite a bit on my phone. I also click wrong buttons when using mouse but that is rarer than pressing wrong button on my phone.
The most annoying part is using touch screen in my car for GPS. It might be bad UI but also viewing angle might be making it hard to press right button. But then again I would not expect a physical keyboard in a car.
In physical land, it is very rare that I press a wrong button. Maybe when playing intense video games, I press jump too soon or use wrong gun. Oh and of course while typing.
I can select very small text on my phone with my bigger fingers. And also with a touch screen you may have more options in a smaller space.
Sure, that may suck for critical things, but there probably are a lot of settings and so forth that could be hidden that are not critical?
But I understand what you meant, I prefer physical buttons as well.