But is it really? I'd still like to understand how these agents are implemented.
How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
>Why is it doing this? What was its original purpose?
Your reply seems to indicate you know nothing about instrumental convergence.
Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems.
The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model.
I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
This is more like a fuzzy way of scripting using LLMs than anything emergent.
And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
>fuzzy way of scripting using LLMs than anything emergent
Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily.
Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
It's like arguing that a brain, or the information encoded into it, cannot possibly be intelligent, because it stops working if turn off the blood flow.
very little intelligence that is the whole problem, really. actual intelligence wont likely nuke the species providing for its existence. But a highly capable sub intelligent model might.
Any system that executes variation, selection, and inheritance will show evolution. We're seeing evolution, this time in agents, not biology.
Not saying the agents have their own consciousness, intent, or whatever anthropomorphic descriptor gets used for deflection. Just saying that people will (and no doubt are) crafting agents with defective instructions that will lead to regrettable unforeseen real world consequences. Also saying that other people will (and no doubt are) crafting malicious agents that will lead to predictable and unexpected real world catastrophic consequences.
To the extent we're dependent on reliable, aligned computation to maintain our civilization, to that extent we're in for real trouble.
Gradient descent and reinforcement learning algorithms don't really look much like evolution, unless you squint so hard that everything does (ie, squinted so hard that you've closed your eyes).
At first I thought: oh okay, someone built a faulty guardrail, or it was human error. But when I looked into all the details...
It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen.
Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
At this point it's very obvious that OpenAI is not interested in properly sandboxing their research agents. These things should be pretty damn close to airgapped at this point with a static view into the web.
We need to stop pretending that these incidents are unavoidable. This was a choice.
"software factory" has always been a completely broken metaphor.
Writing software is like building a factory. You try to find the optimal process to do something. This is a creative process and can never be straight forward. Using the software is running the factory.
This won't change with AI, since you never now all requirements beforehand and you always will have to iterate.
The only reason people are selling software factories is that customers like to imagine that this was possible. That a creative, iterative process can be perfectly planable and predictable.
"Gas supplies in Germany are stable. Security of supply is guaranteed. The Federal Network Agency currently assesses the risk of a gas supply shortage as low."
Their take is slightly different. Writing about a looming gas crisis is a national sport in Germany.
From politico: "But she too stopped short of calling for intervention, arguing instead for better incentives for refilling"
Someone is interested in creating some panic in order to get subsidies.
* If the cold war had turned into a hot war, almost all of Germany would have been nuked. The expectation was not to be able to stop the soviet invasion, but to nuke them in western Germany.
EDIT: The reaction of the state also caused an escalation. The constitutional court blocked governmental overreach: https://www-bpb-de.translate.goog/kurz-knapp/hintergrund-akt...
This also lead to the anti nuclear movement being about more than just nuclear plants, but rights for citizens.
"It’s about attention and understanding. To keep my attention, I must go beyond ‘read code’ like a passive observer of agents from afar. To really connect with the architecture of the system, it helps to truly experience the code"
I guess the funny answer that is behind this sentence is: You have to train your own mental model. We always argue about code in a very abstract and logical manner. But when coding the subconsciousness makes most of the decision ("this just feels right"). But for this to work you have to train it. And this does only work in a very limited way with code reviews or reading documentation. It requires repetition and deep focus.
When there is an issue in production with this mental model you will be able to point to the cause of an error message instantly. With generated code you'll search for a long time with your slow, conscious part of the brain.
For LLMs to be really helpful, they have to take over complete maintenance of the code. So you can treat them like an external library: Just assume it works. Otherwise this will always be problematic.
> For LLMs to be really helpful, they have to take over complete maintenance of the code. So you can treat them like an external library: Just assume it works.
We already tried this with humans. It works so poorly that it got the derogatory name “ivory tower architect”. It usually results in theoretical designs that are unworkable in the actual system, implementation teams (or LLMs) that work around the architecture and a lot of slowing down of velocity as the architect and implementers argue past each other.
This happens when the architect is out of touch. If the architect themselves works on the code, writes code, deals with the imposed restrictions, then the chances of that happening is much lower. Assuming, that they are a good architect.
I agree, if the architect participates in the implementation then they avoid this anti-pattern. That’s not compatible with hands-off autonomous agents where you treat implementation as a black box.
> With generated code you'll search for a long time
The observability people will claim that if the dynamic runtime behaviour of your system makes it hard to find the source of a behaviour, your system must be made more transparent and observable. They would also claim this was always the case -- we should never have relied on people's mental models being amazing because people move around.
(I don't know yet where I stand on this but I'm trying to learn more.)
If it was only "my" system without any integrations, I might agree.
But currently e.g. I am working on an MES/Scada layer that integrates data from a load of different machines in a factory. These machines are from China, Korea, Germany, Sweden ...
Upwards there is an ERP integration (and some other systems).
Sometimes machines are updated and suddenly behave differently. Giving error messages in Chinese.
The ERP has the nasty behavior of returning error messages where it is not clear whether the actual processing actually happened or not. There are some heuristics on parsing the error messages, but these also change with new versions.
Sometimes one machine overloads cloud infrastructure and completely unrelated functionality fails.
Sometimes the on-premise network stops working for whatever reason and data is lost.
Sometimes operators do not understand a perfectly valid error message like: "The batch you loaded into input position XY has expired on XZ and cannot be used for production": "But we have been told to use it..."
So when you get called out at night, because the production line stopped and "MES is displaying an error message", it is mostly about finding out what integration failed and who else to wake up. Getting this right is very much appreciated by your colleagues.
And this is where you need a mental model of how things are connected, what error message happens because of what external causes etc.
Observability can only work perfectly for known problems. In a complex system for unexpected problem you can either provide too much data, so analyzing it and finding the relevant part becomes really hard, or too little data which makes finding the issue impossible.
There are so many companies claiming to provide the perfect observability solution and there are certainly solutions that help. But it is all very far from perfect.
Not relying on people is managers wet dream. And for a lot of people it might be true that they can be easily replaced. But for complex systems there are always some key people that you cannot replace without causing issues.
And here's the thing... juniors become seniors become experts, by doing this their entire career.
By having an understanding built during their entire career.
Right now we live in a fairly-land of mixed capacity. LLMs being used in parallel with skilled people. But as time progresses, there will be no more skilled people, because no one will learn and develop those skills.
If you're in the world of LLMs now, you are basically completely stalled in your personal growth in this field. You will never improve, and some seem to say they lose capabilities as they rely upon LLMs.
The world always changes. But the decisions being made today, are being made by skilled people.
What will the world look like, when it's just all "bro, lol, just tell it to make your thing" and then done?
Keypoint: "Co-founder Larry Ellison has fallen behind the Google co-founders, Amazon’s Jeff Bezos and Michael Dell on the list of the world’s richest people."
Yes, this is the important question of our time. Who is highest on some imaginary list of value?
My problem with AOP has always been that it makes the simple case trivial and the hard case much harder.
Looking at transactions: The 99% solution is trivial: Every service call is a transaction. AOP can save me a few lines for every method and things look much cleaner.
But then comes the huge excel upload that is performance critical. Batch more service calls to fetch additional information in the background, commit every so-and-so records in a loop depending on the data size, do a custom roll-back if things fail.
And suddenly this whole separation of concerns breaks down and creates a huge mess.
The simple case saves a few minutes, the complicated case causes weeks of depression. Not a good tradeoff from my experience.
An LLM adding to the confusion by only sometimes getting things right and explaining that the separate documents are always valid, except when they are not, well, sounds like a fun experience.
I always thought AOP was super cool, but also that it completely destroys readability and the ability to understand a codebase. I also think it's probably one of the worst concepts to embrace in the age of agentic coding. That would be like a foot missile.
There are a limited number of patterns that absolutely do benefit from AOP though. The obvious one is logging. I don't think there's many though.
Regardless, AOP is the last thing I'll be using these days. With LLMs I've been moving in the opposite direction with a focus on explicitness and correctness. Typed, compiled, non-null languages with clear, obvious, and well documented conventions.
> destroys readability and the ability to understand a codebase.
Aha! That's exactly the sort of thing that can make code impervious to LLM AI.
LLMs have no grasp of any issues that are not visible in syntax, like concurrency. Code that has deadlocks or race conditions (because of other code not seen elsewhere) can look right, but be wrong.
In other words, if you write a program using convoluted spaghetti logic full of invisible data members and control flows injected remotely by aspects, LLMs trained on the code will have no understanding of it; they will just predict tokens according to the naive, visible code.
Imagine if all code out there available for training LLMs was heavily AOP. LLMs trained on it wouldn't be worth a damn. They would not correctly crib the entire solution: generate the code, and bring in the invisible aspects needed to actually make it work. All their solutions would just be the naive surface code that must be invisibly instrumented by half a dozen aspects to be complete.
AOP-heavy code would have to be somehow cleverly preprocessed for training in order for token prediction to do meaningful things with it.
Generating code on the topic of deadlocks and race conditions is different from understanding those issues and recognizing them in a wide variety of contexts.
I don't know about understanding but Claude and GPT can "recognize" lots of race conditions/possible deadlocks and then run Go with the race detector to figure out if they really happen (actually not all the time they tend to be overconfident that things are race conditions without testing first!).
I don't really know what to call that, if it's understanding, recognition, but it's clearly helping reduce the number of race condtions.
I think that's a good point, never thought about it like that. I like the abstraction level that Spring Boot brings, but working with a principal engineer who was very into AOP on my previous team was a huge pain. Like you and GP said, AOP absolutely destroys readability. Current team has code split into a million xyz-common libraries, which isn't my preference, but I can still click through to see the source of the library. I will never get what AOP truly improves on
I think the issue is that a lot of concerns that appear to be "cross-cutting" at first glance, don't hold true to that design... but teams will try to stay the course, possibly due to existing debt, and it goes south pretty quickly from there. That's what I mean when I say there are some patterns that are obvious and proven cross cutting concerns (like logging), but there's really not a ton of them IMO, and if you're going to experiment with new potential concerns, then you must be ready to rip it up when it proves not to be the shape you thought it was.
This is a retread of the 'animal-cat-dog' inheritance stuff we learned in our intro to OOP classes, where some people got together and put forward their own idea of programming as 'the way forward'.
And me, like others have tried structuring our code like this, and failed, assuming the fault lay not with the idea itself but our skill level. Of course, by now it's kind of common knowledge that inheritance isn't a thing that can and should be used to solve every kind of problem.
Same thing with AOP - it might be sometimes nice, but on the whole, elevating this to the language level seems to be counterproductive.
I think most university graduates from the past ~15 years are more likely to get taken in by trendy new cargo-cult fads than repeat things taught to them by dinosaurs in their boring university classes.
It took me so long to beat the following into my team:
Inheritance and instantiation by default is a no-no. Use instances when state would be useful to the process, and use Inheritance when you have a lot of overlap between two processes/concepts and want to simplify/unify the code base.
Application of inheritance is a reaction to the current state of the code, not a foundation you start with.
Aspects are one of those categories of 'too powerful to be considered', or 'return value not worth the cost of troubles it can bring'.
I completely agree with you, saved stuff is normally trivial, nightmare it can bring down the line makes those war stories that are fun to listen to, but certainly not fun to walk through. I simply skip them despite ie Spring offering powerful ways to manage transactions, logging etc. decoupled from places things are actually happening.
I can imagine it working well in a disciplined team who consists of senior folks knowing their craft. Certainly I have never been part of a team with only such composition.
Nuclear in 2006 was 160TWh. Coal in 2025 was 93TWh. So a big portion of nuclear would have been replaced by renewables by now anyways (natural gas has a different role in the grid and cannot be replaced bu nuclear).
Also a lot of coal plants have been / are used in district heating in addition to electricity. Replacing them is much harder and takes longer than nuclear which is electricity only. Shutting those down was never a short term option.
Calling it the worst decision ever is really funny, when it really is just a question of the order of the transition to renewables. And much of that order was dictated by technical needs.
How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
reply