It's not as specious as you believe. Your bank can almost certainly call your mortgage loan at any time. From the other side, prepayment of a loan, even without communication, happens all the time.
It's been a while since I read my mortgage note, but I believe the only condition under which it is callable is a default. Unless I've misunderstood what callable means in this context.
You've got the right of what callable means. After researching more about this, I have learned that some mortgages also have additional callable acceleration provisions to lower the bank's risk - such as when a borrower's credit score falls.