I'm not opposed to "move fast and break things" but our problem is that's the only lever we pull. For every "... and break things" there needs to be a phase of "clean up, everybody do your share". It seems the modern development framework is allergic to cleaning up. There's so many excuses given but if you don't clean up you can't move fast.
In physical reverse engineering there's a common pattern people use: buy 3. One to break, one to modify, one to reference. You need the one to break because you're going in blind. The problem has a lot of unknown unknowns. It's often difficult to take things apart (especially these days) without breaking them. But the second time it is much easier to do nondestructively.
But I'm also a big fan of taking time to think and understand. To gain deep understanding of things. I've always found this to be helpful and allowed me to move faster in the long run but I often face resistance to this because everyone wants me to "move fast".
The problem is I think people have the illusion that you can run a marathon by doing consecutive 100m dashes. It sounds nice in theory but I think there's no surprise that burnout is at an all time high and things are getting sloppy.
It's weird, we've systematically created a work structure that has the same principles as scams: frame everything as an emergency so the mark doesn't have time to think. Why the fuck are we scamming ourselves?
> It's patently insane to demand that humans alter their behavior to accommodate the foibles of mere machines
I don't think it's insane, we do it all the time. Most tools require training to use properly. Including tools that people use every day and think are intuitive. Use the can opener as an example (I'll leave it for you all to google and then argue in the comments).
The difference here is that this tool is thrust upon us. In that sense I agree with you that the burden of proper usage is pushed onto the user rather than incorporated into the design of the tool. A niche specific tool can have whatever complex training and usage it wants.
But a general access and generally available tool doesn't have the luxury of allowing for inane usage. LLMs and Agents are poorly designed, and at every level of the pipeline. They're so poorly designed that it's incredibly difficult to use them properly and I'll generally agree with you that the rules the author presents aren't going to stick. The LLM is designed to encourage anthropomorphization. Usage highly encourages natural language, which in turn will cause anthropomorphism. The RLHF tuning optimizes human preference which does the same thing as well as envisaged behaviors like deception and manipulation along with truthful answering (those results are not in contention even if they seem so at first glance).
But I also understand the author's motivation. Truth is unless you're going full luddite you're going to be interacting with these machines. Truth is the ones designing them don't give a shit about proper usage, they care more about if humans believe the responses are accurate and meaningful more then they care if the responses are accurate and meaningful[0]. So it's fucked up, but we are in a position where we're effectively forced to deal with this.
So really, I agree with you that this is insane.
> I don't have a proof, but I believe that "AI safety" is inherently impossible, a contradiction of terms
To paraphrase my namesake, there's no axiomatic system that is entirely self consistent.
Though safety and security is rarely about ensuring all edge cases are impossible, but rather bounding. E.g. all passwords are hackable, but the failure mode is bound such that it is effectively impossible to crack, but not technically. (And quantum algorithms do show how some of the assumptions break down with a paradigm shift. What was reasonable before no longer is)
[0] this is part of a larger conversation where the economy is set up such that people who make things are not encouraged to make those things better. I specifically am avoiding the word "product" because the "product" is no longer the thing being built, it's the share holder value. Just like how TV's don't care much about making the physical device better but care much more about their spyware and ads. Or well... just look at Microsoft if you need a few hundred examples
> After all, medicine is all about knowledge, experience and intelligence
So is... everything?
LLMs are really really good at knowledge.
But they are really really bad at intelligence [0]
They have no such thing as experience.
Do not fool yourself, intelligence and knowledge are not the same thing. It is extremely easy to conflate the two and we're extremely biased to because the two typically strongly correlate. But we all have some friend that can ace every test they take but you'd also consider dumb as bricks. You'd be amazed at what we can do with just knowledge. Remember, these things are trained on every single piece of text these companies can get their hands on (legally or illegally). We're even talking about random hyper niche subreddits. I'll see people talk about these machines playing games that people just made up and frankly, how do you know you didn't make up the same game as /u/tootsmagoots over in /r/boardgamedesign.
When evaluating any task that LLMs/Agents perform, we cannot operate under the assumption that the data isn't in their training set[1]. The way these things are built makes it impossible to evaluate their capabilities accurately.
[0] before someone responds "there's no definition of intelligence", don't be stupid. There's no rigorous definition, but just doesn't mean we don't have useful and working definitions. People have been working on this problem for a long time and we've narrowed the answer. Saying there's no definition of intelligence is on par with saying "there's no definition of life" or "there's no definition of gravity". Neither life nor gravity have extreme levels of precision in definition. FFS we don't even know if the gravaton is real or not.
[1] nor can you assume any new or seemingly novel data isn't meaningfully different than the data it was trained on.
> [0] before someone responds "there's no definition of intelligence", don't be stupid.
Way to subdue discussion - complaining about replies before you get any.
But you're wrong, or rather it's irrelevant whether something has intelligence or not, if it is effectively diagnosing your illness from scans or hunting you with drones as you scuttle in and out of caves. It's good enough for purpose, whether it conforms to your academic definition of "having intelligence" or not.
If you want to be dismissive and with quick quips that's not a discussion. There's plenty to respond to without relying on "there's no definition of intelligence" and definitely not "so I'll just make one up".
> or rather it's irrelevant whether something has intelligence or not
But it seems like you want to be dismissing, not engage in discussion.
> whether it conforms to your academic definition of "having intelligence" or not.
Why pretend like I don't care that it works? In fact, that's the primary motivation of making these distinctions.
Yeah, I mean, I don't know where all of this is going, but I do think that the ancients cared WAY more about "embodied knowledge" than we do, and I suspect we're about to find out a lot more about what that is and why it matters.
There's a lot of definitions of bodies. Though I'm unconvinced one is needed. A brain in a box is capable of interacting with its environment far more than such a thing could even a decade ago. Is it the body or the interaction?
As we advance we always need to answer more nuanced questions. You're right that the nature of progress is... well... progress
Everyone is crazy, just most people are afraid to admit it to others. A lot of people are even afraid to admit it to themselves. Some people pretend so long they forget they're pretending. But wouldn't that itself be crazy?
Having taught at a university I'll say that the general reason is because there's already too much to teach, so you do your best. It's extra hard since there's a million people saying "why don't they teach X?" and you have to accommodate them.
There's problems like do you teach Python or C? It sounds silly but the difference is not about languages but how much you teach about systems. Teaching Python you get people going and they can produce faster, which does help students get less discouraged. But teaching C forces learning about the computer system and enables students to dive deeper to teach themselves many different subtopics that no 4 year program can.
What I think is generally missing and would be good to implement is code review and teaching how to understand a large existing codebase (all that grep, find, profilers, traces, tags, and all that jazz). This often gets taught in parallel (e.g. have students review each others code) but it's hit or miss, a lot of extra work, and not everyone does it.
Here's the shitty part: I was often told by peers and people higher up "don't look at student's code, just look at output and run tests." I always ignored, because that advice is why we're failing so many students. But I also understand it because professors are overburdened. There's too much work to do and teaching isn't even half the job. Then every new administrator or "office assistant" they hire, the more work you have (seriously, it'll take days to book a flight because you have to use some code but it takes 2 days for someone to tell you the code and 5 more to tell you that it was the wrong code and it's clearly your fault because you clicked on "book flight" and not trips > booking > flights > schedules > trips > access code > flights > search available flights. Honestly, I think all this llm agent stuff would sound silly if people actually just knew how to design things...)
I think it is a real increase in the rate of detected attacks, not just awareness, but whether that’s an increase in vigilance or an increase in attacks is hard to know. I suspect both, of nothing else because awareness drives both vigilance and attackers inspired by the earlier attacks.
I looked pretty hard, with some LLM assistance, so if it was "are we just hearing about it more now" it would have to be old attacks that happened without being discovered and written up.
Is that true? I think for it to he true we'd have to overly abstract the definition of technology to the point of uselessness.
You can draw images in the sand. Is a stick "technology"? What about using your finger?
Do we need paints? There are natural dyes. I don't mean in the sense of extracting things but some are as simple as "smash this berry". I believe the answer to this is rather critical since you specifically mention cave paintings. Many of those were done by hand, not by brush.
What about things like rock balancing? Sand sculptures? Singing/vocal instruments? Poetry (spoken, not written)? Story telling (ditto)? And so on
There is so much we consider art that can be done by any human with no tool use nor any external objects. I won't even mention how people call a sunset a work of art, and I do think we should avoid that as it has the same problem I bring up with defining technology. But I do not think most people would consider speech or vocal sounds technology, though certainly we would include things like writing.
It takes a considerable amount of development before you can make the distinction at all between separate concepts of art and technology. For a long time there wasn't a split because it was difficult to conceptualize how to split the two.
Those are dark patterns and people are not aware of them. It is an external actor trying to take control of your agent.
I don't think it's necessarily wrong to have those prompts, but it is if it's hidden or obscured. Intent matters a lot here. Which the response to name shaming (and how you name shame) is actually the important part. Getting overly defensive is not the appropriate response. Adding clarity and being more transparent about why such a decision was made is the correct response. We're all bumbling idiots and do stupid stuff. But there's a huge difference between being dumb and malicious, even if the outcome is the same
In physical reverse engineering there's a common pattern people use: buy 3. One to break, one to modify, one to reference. You need the one to break because you're going in blind. The problem has a lot of unknown unknowns. It's often difficult to take things apart (especially these days) without breaking them. But the second time it is much easier to do nondestructively.
But I'm also a big fan of taking time to think and understand. To gain deep understanding of things. I've always found this to be helpful and allowed me to move faster in the long run but I often face resistance to this because everyone wants me to "move fast".
The problem is I think people have the illusion that you can run a marathon by doing consecutive 100m dashes. It sounds nice in theory but I think there's no surprise that burnout is at an all time high and things are getting sloppy.
It's weird, we've systematically created a work structure that has the same principles as scams: frame everything as an emergency so the mark doesn't have time to think. Why the fuck are we scamming ourselves?
reply