If someone breaks into my apartment, I have more problems than losing my Yubikey. And my threat scope does not include agencies that would get my passwords AND be able to break in and steal my key. And if it did, I would think they could just take it from me.