I don't see how it's less secure than storing passwords in ~/Documents/Passwords.txt. And it's a second factor, so combined with first factor the result is pretty secure. You can't browse other people phones, even without security enclave.
Websites running javascript weren't supposed to browse other people's computers either, but we all know how that assumption went. Yes, it has gotten better the past few years, but the whole point of a security key is that it takes a purpose-designed piece of hardware and software, with a minimal attack surface. A phone is far from it.
