It's not about being reasonable, it's about having a good reason to kick basically everyone out who isn't 100% Japanese, or who has enough money to prop up their aging population and get almost nothing in return.
It might also be just correlation? Like, those agents were all instances of the same one or two models, so if that model has a preferred order it tries finding vulnerabilities in (the same way all current models have a particular writing style baked into them by RLHF), then most of the swarm will follow the same order and converge on the same services to exploit.
Toyotas aren't just for show either, like they are the top of their game 4x4s, they sell everything from utility vehicles for mining and military use, all the way up to luxury models.
The public models won’t hack because they have a classifier that shuts down anything that looks like hacking; without the classifier they are perfectly capable of hacking, multiple third-party evaluators have confirmed this.
The models were not trained on systems intentionally connected to the internet; they chained mutliple zero-days (that they discovered) together to get access to the open internet and into huggingface.
> The models were not trained on systems intentionally connected to the internet...
If Amazon connects an AWS Top Secret region to the Internet, it doesn't matter whether or not it's intentional... they're getting nailed to the wall by the US government either way. Frankly, it's way worse for them if it was accidental; deliberate, sophisticated sabotage is a much better story than rank incompetence and/or negligence.
A similar sort of thing applies to the manufacturers of tools that they claim to be dangerous, that have been deliberately built to exceed their authorized access to other computer systems, and are deliberately being tested on how well they can do the thing they've been built to do.
Deliberate, sophisticated sabotage by one or more humans in their employ is much more forgivable than "Whoopsie, we didn't think to make it literally impossible to connect this dangerous automated computer-hacking tool to the Internet.".
It seems that we’re mostly in agreement? I agree that OpenAI has been terribly irresponsible, and that this attack being an accident makes things worse.
What I dispute is that AI agents are simple tools. I think rogue is an accurate word to describe them; I think what OpenAI is doing is more akin to gain-of-function research on a dangerous lifeform. I think this attack would have been prevented by air-gapping, but that wouldn’t solve the fundamental issue which is that they are creating something dangerous that they have no idea how to control
> What I dispute is that AI agents are simple tools.
You're in luck! I agree that they are not simple tools. I never claimed that they were. Slow down and read more carefully.
I couldn't disagree more with the insinuation that the LLM manufacturers are doing things akin to scary research on uncontrollable hazardous biologicals and with the claim that "rogue AI" is the correct thing to call those complicated tools. The first is fearmongering which I'll address indirectly in my second-to-last paragraph. The second shifts the conversation from
"How could you have not predicted that the computer-attacking tool you built, explicitly instructed to attack computers, [0] and connected to the Internet attacked someone else's computers that were connected to the Internet?"
to
"Wow, that thing went rogue. Noone's to blame but the tool, and it can't be blamed!".
There are so many extremely complex systems out there [1] and when they do things that we don't want them to do, it's not described as "going rogue"... either there's some error(s) in the underlying system that caused the confusing behavior, or the programmer didn't understand well enough how that system works.
> ...they are creating something dangerous that they have no idea how to control
Ignoring the fact that "put it in a box and don't let it out of the box" is the simplest possible control mechanism, [2] if they have no idea how to control the tools they've been building, it's because they haven't bothered to learn as they went. Tangentially related, there's a Tumblr post I saw recently that's a fictional conversation with the Tumblr user and the CEO of Anthropic. It went something like
Amodei: We're building an incredibly dangerous tool that has a 10% chance of killing all humanity. We *must* be regulated to ensure everyone's safety!
Tumblr User: Regulation takes time, please stop building the incredibly dangerous tool?
Amodei: ...No.
[0] That is -after all- the task that the tool was put to when it attacked other people's computers.
[1] Have you ever tried to really understand a specific AMD x86-64 CPU, let alone the entire stack that makes up the system that is a consumer-grade PC and its installed software? Both are definitely way more than any one human can keep in their head at once, and are tasks that would take a very long time to complete.
[2] ...it's also the most appropriate control mechanism for the task that started all this conversation, and neither of the major manufacturers used it!
reply