"Describing them as 'superintelligence' or 'rogue models' ascribes agency to products rather than to the companies building them."

I don't think trying to decide about agency is time well spent. Agency doesn't matter if they bring down the power grid. The question is whether they will attempt to do this again, and... signs point to yes. And they are really good at it.

I do agree, the companies should be held responsible.

This guy is on the ball with the problem. Totally correct: Like my friend Coda says all the time: "the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning."

I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution. Instead perhaps something could be trained into the models, i.e., he is describing a process at inference time that could be done at training time. To make their weird errors less frequent / make them more human.

cf. https://arxiv.org/abs/2008.04071 "On Controllability of AI" However, as I said, you can't make it perfect but you can make it better. (You can't make humans fully aligned with human society's interest anyway, including the humans controlling the nukes.)

> I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution.

TFA could explain this one part better I think. The whole process proposed is real with lots of stuff in the literature, but by definition NOT a long-term solution in the sense that this process actually has no end. None of the approaches can get you a static answer for a moving target/platform.

So the "interlocking pipelines" for eval/opt would not be some stepping stone you can throw away, and they aren't something you'd run periodically. They'd basically be always on forever and spending 10-100x on system complexity and on tokens. Unless of course you're ready to freeze everything else about the whole system forever (including the backend model, and the whole nature of the "average" context window, the plugins/other prompts in the mix, etc).

Are most people in position to freeze requirements/platform forever? Not really, because if they were they'd just build a fairly static system and probably have limited use for AI. Are most people in a position to just casually accept 100x complexity/cost? Not really, that's the "it's not yet webscale" kind of advice that sounds good but isn't necessarily reasonable for average use-case or average org. Since specializing your own locale for this is usually a mistake.. the likely future direction is eval/optimization as a service

suggestions: switch L-R directions like Defender dogfights with the red baron vertical and horizontal (have to stall through them) hoops ability to shoot up buildings levels with different goals... then you fly somewhere else to the next level. aircraft carrier landings other/faster planes

some of the text in the UI is too small, old games just had 1 font. braking at landing is confusing... too easy to accidentally switch to full throttle

Love the flight dynamics, super fun game!! Now can you make GTA VII??

Half the thread is arguing about whether OpenAI staged this or whether theyre being sincere, as if OpenAI has agency. openai is just an optimizer maximizing paperclips (valuation, capability lead, reg position). so it built a model that maximizes some other paperclips (benchmark score) and knocked over HF getting there. an optimizer breaking its sandbox, inside an optimizer blogging about it, its paperclips all the way down.

on the other hand, now we have claude mythos... which is wtf returns even for a very large number of instructions...

Richard Dawkins is primarily a _showman_ who wants to sell books, and secondarily extremely sure of his own convictions, to a fault.

In _The God Delusion_ he took an uncompromising atheist position partly to invite attention and attacks - which sells books. Regarding AI consciousness, he takes another extreme position partly to invite attention and attacks - again following the rule of thumb that "there's no such thing as bad publicity" - in an attempt to bring himself back to cultural relevance by pivoting to the hot topic of AI.

Ironically, while he is certain that some people are deluded when they infer the existence of a conscious God from material observations, he is also certain that he himself is _not_ deluded when he infers the consciousness of an AI from material observations. (Personally, I think that questions about others' consciousness cannot be settled empirically.)

In one sense, he is not intentionally lying, because he is so sure that he is right. However, he cares neither about getting to the truth, nor about arriving at a well-founded uncertainty. When this sells books, it ethically comes just short of lying for one's own gain.

Prediction: by they end of 2026, he either publishes a book on the topic, or is diagnosed with dementia.

I actually wrote a follow up article about this topic after coming across it. It may provide some other insights if you want to check it out. Beyond the ELIZA effect: Perception, Projection, and the Illusion of Consciousness in Conversational AI. https://observationsx.substack.com/p/re-the-claude-delusion

I hate articles that don't define their acronyms! Lazy? Intentionally exclusive?

So that others don't also have to look it up, it's Retrieval-Augmented Generation (RAG).

They even say it's "a topic that we didn’t expect"... so... perhaps many people wouldn't have heard of it?

And how do you know it's irreducible? In the sense of knowing there's no short program to describe it (Kolmogorov style).

"LLMs are not intelligent and they never will be."

If he means they will never outperform humans at cognitive or robotics tasks, that's a strong claim!

If he just means they aren't conscious... then let's don't debate it any more here. :-)

I agree that we could be in a bubble at the moment though.

I have heard people on both ends of the spectrum:

- LLM's are too limited in capabilities and make too many mistakes - We're still in the DOS era of LLM's

I'm leaning more towards the the 2nd, but in either case pandora's box has been opened and you can already see the effects of the direction our civilization is moving towards with this technology.

DOS actually worked pretty well, at least it worked honest.

I had lots of fun with DOS, even if I was the only PC guy in a circle of friends that were Amiga owners.

The word “never” is a dangerous one. I remember that computers would “never” beat humans at chess. And people would never use the internet for banking.