The argument is that without your phone, you likely have no recourse to stop the attack. Since your passkey on the phone is what controls your access, now.