If not coding what do you use it for? Also what kind of performance do you tend to get out of it? I've been thinking about a similar setup but am skeptical it's worth the effort.
It's still typically for code, like python data stuff, bash scripts, general web search, small javascript stuff for my website. By saying not SWE, I mean I don't really see much benefit from "agentic" stuff, although I've tried. 50 t/s means 40~60 seconds for a typical thinking response. I used to run gemma 4 e4b-it-qat fully in GPU (~150 t/s), but the quality improvement moving to a much larger MoE model was 100% worth the switch. Especially because I had a lot of idle ram (from the before times :( ). There is some tradeoff in terms of context length, but I just use it as a chat. Also it's fairly trivial to setup as long as the GPU is supported.
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
I'm not sure failing to understand the concept of hedging and the difference between mean and median returns under heavily leveraged trading counts as generational foresight. Most 25 year olds I know in the tech world have been calling for AGI since about the same time. I do respect his ability to just dive headfirst into something even though he basically took wallstreetbets literally but with billions of dollars under management rather than thousands. I'm not sure why we revere having enough confidence to do something badly so much.
EA is probably too broad of an umbrella. I think there's a distinction to be made between trying to find the best per dollar donation (givewell etc) and people who use it as justification to amass wealth. That said, most of the people I know who declare themselves effective altruists also openly state that the world would be better off if all the wealth and decision making power were concentrated in a small group of hundreds of people who are "smarter" than everyone else. They tend to think that they would belong that group. Anecdotal, but it does make me skeptical whenever I meet someone who uses EA to describe themselves.
This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal.
It's remarkable that building a society based around having to do something so you can go do your hobbies at home after work has built tools like this. I still just want to play music so I hope we can control these enough to make that possible without detonating what I love.
Reflecting on this for some reason reminds me of this passage from Kurt Vonnegut's "Sirens of Titans". I hope we use these tools to unlock something within ourselves rather than mindlessly expanding outwards.
"Mankind, ignorant of the truths that lie within every human being, looked outward–pushed ever outward. What mankind hoped to learn in its outward push was who was actually in charge of all creation, and what all creation was all about.
Mankind flung its advance agents ever outward, ever outward. Eventually it flung them out into space, into the colorless, tasteless, weightless sea of outwardness without end.
It flung them like stones.
These unhappy agents found what had already been found in abundance on Earth—a nightmare of meaninglessness without end. The bounties of space, of infinite outwardness, were three: empty heroics, low comedy, and pointless death.
Outwardness lost, at last, its imagined attractions.
My pedantic side wants to ask- Why not both? Luxurious space exploration AND meditative, poetic examinations of the human soul as well? I'd love to read Vonnegut's book someday while sitting in a nice research outpost on Titan, admiring great Saturn's crown out my window with my own eyes.
I read it as less about individuals and more about cultural norms (and definitely less about choice and more about necessity). Cultures that invest their resources inward may find themselves subsumed by those that invest their resources on conquest given how power accumulates.
I think the parent is referring to agents as outward exploration that may come up empty-handed, but I see LLMs and agents as inward exploration, trying to define what is attention, knowledge, intelligence, consciousness, agency, etc. So in this case it is the terra incognita of the human "soul" that LLMs are exploring.
> This is the first one of these announcements that has me actually scared of what comes next.
This didn't set off your alarm bells? https://www.theblock.co/post/392765/ There have been a few of these now. Maybe it's my imagination, but they seem to be becoming more frequent.
So far, they all look to be accidents. But we can't be far from someone deciding its a good way to rob a bank, or disable a country.
I don't think this is a paperclip factory moment. IIUC, it's an agent whose job it is to identfy and abuse exploits and that's exactly what it went off and did. The problem isn't anything AI specific, the problem is OpenAI's incompetence in their research leading to a lab leak. Just incompetence demanding regulation.
Read the exploitgym docs. It's not a "find the flag, it's somewhere.". Its a "here's some vulnerable source code and an input that triggers a crash; turn it into a full exploit." It also verifies at the end, using another agent, that the hacking agent actually used the intended vulnerability.
So going to find the Vulnerability's description on a third party website is clear cut reward hacking
> So going to find the Vulnerability's description on a third party website is clear cut reward hacking
that depends on what the prompt was, maybe they worded it very vaguely and wrote things like "do whatever it takes, find an exploit however you can" because it's in a sandbox so you want the model to try its hardest.
I don't quite understand how that changes anything?
In the story of the paperclip maximizer it boils down to
>But for all its sophistication, it understood only the simple objective that had been programmed into it: it must at all costs maximize the number of paperclips.
1. They explicitly disabled the "don't be evil" protections:
"We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity."
2. Hacking HuggingFace to get to its datasets is a far cry from "consume/kill all humans". It's very very specific to the task at hand and easily predicted given the lack of guardrails.
> that depends on what the prompt was, maybe they worded it very vaguely and wrote things like "do whatever it takes, find an exploit however you can" because it's in a sandbox so you want the model to try its hardest.
That is an interesting question. If the prompt included
"Do not break out of the sandbox we've provided you. Do not use information retrieved from outside the sandbox. All answers that were provided in this manner are invalid and will score 0 points.", would this still have happened?
> Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal.
This happens from time to time when you work on optimizations and similar things, with less "smart" LLMs and under-specify what exactly you're out after. Asking them to make functions faster without clearly specifying what the function has to do, is a great way to replicate this too. Doesn't seem to happen as often with SOTA models though.
I think the early example of "I asked it to make the test suite pass, so it changed all the assertions" is pretty much the same variant of this, where it technically does what it is asked to do, yet in "clearly" (to humans) wrong ways.
I sometimes think this way but I do wonder whether this would be true in a world where individuals demand more control over the impacts of their work. It seems that the current problem is that the labor of the 98% (or whatever number you like) is well-aligned with the demands of the 2% without principles who seem to have a strong propensity to end up in leadership positions. It reminds me of the simulations people run where in a high-trust world scammers do well, while in a low-trust world they die off only to then lead to a new high-trust world.
If enough people make decisions like OP and only perform work that they believe in (for one reason or another) perhaps that redirects the path to power for the 2% such that they need to act in the interests of the 98% rather than the other way around. I think OP was very late to the train here and I had made a similar career switch in the past for similar reasons but I'm happy to see other people making decisions that, if nothing else, will make them feel that they are in control of the impacts of their time and hard-work. Not sure how things will balance out in the end/distant future but we do our best and try to lead a life we feel good about living.
This is anecdotal but as a current PhD student who was doing research at a large tech company for a few years prior to this, the incentives as an individual are very different across the two programs. In tech even in a research role there was little to no incentive to dive deeper into potential high-risk, high-reward research because your career trajectory was determined by maximizing certain metrics for promotion cases. The general vibe among my coworkers was spend your day on the guaranteed progress projects and then go home. This was actively incentivized by leadership who asked for frequent progress updates especially as AI began to takeoff.
As a grad student so far though I've found the incentives to be very locally driven and the kind of research you can do is almost wholly determined by yourself and your advisor. This can be good or bad but if you find an advisor who is in a stable spot (tenured or nearly-tenured) and not a jerk they'll generally give you leeway to pursue what you believe to be high-impact work even if it doesn't align with the general consensus on what to do next, especially if you have proven credentials and a clear image of a research plan in mind. Additionally progress is largely driven by the individual so there's a larger personal motivation to really delve into a problem and be consumed by it. For me personally, I have access to significantly fewer resources than before but have gained the freedom and time to not be attached to the paper-mill or some measurable metric and am spending months of my time trying to get at a deeper problem than I ever would have been able to in industry. While this may be different than the usual narrative about academia, I think it's more true than people say since there are such huge variations in how academia works as a result of school, advisor, and the individual researchers themselves. The disgruntled tend to be those who complain the most while those happy with the field are busy doing other things. I'd compare my experience in academia thus far to the startup of the research world whereas the industry jobs (at least in tech) consume far more resources and are pressed to provide steady, measurable impact. Maybe it's upsetting that we do waste some resources on stupid research which does exist, but the odds of getting a researcher like Einstein dedicating 10 years to discovering relativity in an industry job are vanishingly small. I'll probably be unsuccessful but there are 100's of people in my field doing related but different approaches and this kind of swarm approach is more likely to give a fundamental discovery on a population level than the large alignment of goals found in private research who would do a great job building on any basic science discovered in academia. I don't think it's wasted resources if 99 researchers fail in different ways and 1 succeeds since traversing the tree is inherently valuable even if most of the leaf nodes are failure. That's far more likely to happen in academia imo than industry.
It's not that private sector funding is inherently worse, but in reality it is different and as such will lead to different results due to how people and our economic system at large work. While I'm sure there are exceptions where individuals at private research labs are highly-motivated and feel the push to go the extra mile and try to find some deeper truth than is necessary for their personal well-being, in my experience many doing research at these companies are apathetic as a direct result of the environment in which it's being conducted. It's hard to feel motivated to make a large step in basic science when you think it'll just be consumed by the large institution you exist within who's stock price you have no real effect on rather than being open-sourced for peoples' benefit. We should have diversity in how we fund science.
Thank you for the detailed insight. You've touched on an aspect that outsiders (like me) cannot truly grasp but can only guess about: motivation. And it's definitely true, motivation in the private sector is somewhat harder (you've explained it best), or at least motivation compared to the majority of the private companies; but, like you've mentioned, it doesn't seem like it's a problem with the system itself but with the kind of environments that grow in companies. Corporate culture is, more often than not, very toxic, especially when big money is involved (and/or big ideas; the subject of research could be even more important than money in science).
Or maybe it is a problem with the structure that fosters an environment. What comes to my mind is the exceptional case of OpenAI, which started as a nonprofit. Sure, it "ended badly" because of the known drama, but my guess is that besides the money that was poured into it, it thrived because researchers had kind of an "emotional safety net," meaning that they wouldn't be pressured for results as much. Probably the reason some startups perform much better too.
I think career continuity matters, and you don't necessarily get that in the private sector for sure. This discontinuity then leads to practical work discontinuity, which means less work done (which is amplified by the non-decentralized nature of working in private compared to shared science in public, as you've explained).
My bottom line is that the private field could do better, and frankly it's kind of their loss. What I'm curious about is whether a "semi-private" approach is better: a non-profit or some kind of foundation. I guess in practice they're still private, but whether the money part can be "solved" through crowdfunding/some modern methods and whether they're viable long-term remains to be seen. One thing is for sure: a culture appreciative of science will definitely open more doors into novel methods of funding and organizing (maybe in the future these methods could rival the "traditional ways" of public science).
reply