I think people also worry about the SIM card swap attack. Even if you have multiple ways to authenticate, RSA token, MFA app, and sms auth, the SIM card swap makes it way easier for someone else to have the thing only you should have.