This is why I have a completely separate private key that I generated for my phone, using a different password that's unrelated to the passwords I use for all my other machines' private keys, and I use password protection on the phone itself to thwart casual intrusion. If I lose my phone, it's trivial to remove the phone's public key from my servers' authorized_keys files, at which point the phone can no longer be used to log in.