These are all silly non-problems (speaking as someone who works with lots of professors around how to put AI systems to effective use)
-the thing that motivates you doesn't have to be the same thing that teaches you (e.g. a book can be a useful learning tool even if it isn't human)
-you can absolutely choose to interact with an LLM in ways that generate productive struggle. If you naively ask it for answers it won't be much use in this regard. But that's just a poor choice of how to use the tool. Just like hitting yourself on the head with a book doesn't mean the problem is with the book as learning tool.
-You don't need structured content 'on long scales' to learn. You learn a little bit at a time. Long textbooks are a side effect of the economics of moving paper around. Not a prerequisite for learning. An LLM will happily mimic the table of contents of a textbook or an outline for course. And you can use future prompts to generate the bits you need as you need them. One-shoting a textbook is a silly use of the tool. Plus, we're not short on textbooks. We're short on ways students can get customized help: answers to the specific ill-posed questions they have in their heads. LLM's are quite good at this. You can have textbooks AND LLM's (and professors)
-LLM's are happy to disagree with you. Just ask them to and they oblige. Ask them to respond as if they were a tutor and they will mimic the assuredness if appropriate.
- Only highly self motivated learners can learn from a book. Most students needed lecture and office hours.
- Students (Humans) aren't going to struggle on average. They're always going to take the path of least resistance. I know of some successful algebraic geometers who did indeed hit their head on heartshorne in grad school.
- a) Structured content generated on long scales is way more effective because you can tie together many details. It would be weird to learn from chapters, each unrelated to each other in style and content.
- b) We're actually not short on students getting help. Office hours were abound at my university. I actually found that customized help was detrimental; if a TA answered every question a student had, the student wouldn't put the effort into trying to learn the material. Imo you want to see students struggle. I've seen students go from F to A- by just forcing them to struggle on the content.
- Not really. I discuss complicated math and programming with models regularly and when I say something like "actually you're wrong" they say "yes, you're right". If I ask them to be confident, they will not change their wrong answers. Humans have a much better conception of self and opinions than LLMs
Spoken as someone who has never seen a weather forecast. Computers have long produced results that were, if interpreted naively, incorrect. These systems require humans to take into account the conditional, probabilistic nature of the output and the inherent limitations of the model the computer was executing, when they interpret the output. LLM's are no different. They give you next token prediction probabilities. That is all. There is no particular reason to think that comes (or even could come) with a promise that those tokens will express truths about the world and only truths. The fact that greed-driven hype, and our human inclination to anthropomorphize the token generator inclines us to imagine things should be different doesn't make it so. No model output is truth. Never has been.
I'm genuinely unclear on what you mean here. Do you mean:
1. Most mathematicians are currently largely paid as part of their academic job (as a faculty member) and will continue to do that work (including the research part) even if extra-mural grant funding goes away. That seems entirely plausible to me.
Or do you mean?
2. Real mathematicians will continue to devote a significant part of their life/energy to mathematics research as a hobby, even if no one is paying them (in any way) to do it? That seems more of a reach and makes me wonder if many currently folks current employed as mathematicians aren't 'Real mathematicians' from your point of view.
So we should return to a renaissance world where only the wealthy or sponsored can do research, teach, and promote the wonders of creativity. Just hope the oligarchs give us prols enough gruel to create our own solipsistic knowledge no one else knows much less cares about, entertained by LLMs optimized to placate our desires and extract money/attention.
A world of cheap AI makes research available to anyone who can afford to run the AI, in a similar way to how ubiquitous cheap computing hardware has made software development possible at the personal level.
As a sometime experimental scientist and occasional manufacturing engineer, this is hilarious!
Discoveries for abstract theory, entertainment, and perhaps regurgitating media, or even basic education can be done with the help of LLMs, but you can't make anything. You can't test anything. You aren't living in the real world. You'll just be lulled into the false belief that everything is simple and fine and beautiful, just as you always wanted, and as airplanes fall from the sky.
Hey, maybe it could work on Cosmology, if you gave it access to the data?
I'mn talking about math research. Of course you can test things: does the formalized proof check? This is what makes automated math such an attractive target for AI.
As someone said below, many research on the side for a decade while having an ordinary job/life. It's often the best environment. No great research was reached at IAS.
With the assistance of AI, we are entering a golden age of individual math research without the need of huge institutions
Uh, John von Neuman didn't discover anything of significance at IAS? Just Hilbert Spaces and Computer Architecture, forget String Theory or Godel's Incompleteness, LOL!
New best prompt:
Write me a really cool math proof that will make me lots of money fast!
Why not let it just spit out the next token on its own without interference, if you can barely fool yourself into believing you understand what it proves... or hallucinates.
No, we should advance to a post-scarcity world where everyone can do research, teach, and promote the wonders of creativity because no-one has to work for a living.
I don't expect Sam Altman and his ilk to deliver such a world to us, but the problem is Altman, not the tech itself.
If what you're trying to build is a perfect match for the full set of abstractions React provides then it's gonna be cleaner to use React than have AI re-invent it. But most cases I run into don't fit that description.
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
Respectfully, go build one, including doing RLHF and RLVR. Those phases generate lots of tokens, then get scored on the entirety of the output, then optimize based on a scoring of that output. It doesn't check a "prediction" against what was actually "next" in data, because there isn't any "next token" data it's training on.
It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.
Your mistake was assuming people would be bothered to understand the details of how things work. Most people are lazy and don't know the details of how anything works.
My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.
Many people are in fact claiming the thing you are saying they are not - even if you are not. The reason they are claiming it is that it was true at one point, and most intro courses/blog posts/videos still describe them that way and then hand wave some "other stuff at the end". You can even see a comment here that refers to the gpt-2 paper. LLMs were trained to predict the next token, produced a distribution to do so, were scored against their prediction v the truth, and the weights updated so that the probability distribution made it more likely to predict the truth from that sample next time. They were, in every sense of the word, a next token predictor.
They are no longer that thing due to post training. They simply aren't making a prediction, and they aren't even optimized for the next token. If I give a distribution of the heights of the population, I'm not giving a prediction either. Distributions don't imply predictions.
Why the desperation to hang onto the word "prediction"?
> Many people are in fact claiming the thing you are saying they are not - even if you are not.
My original comment said “no one here.” Please show me where someone in the comments here is claiming that.
> Why the desperation to hang onto the word "prediction"?
No desperation here. It’s just a word that conveniently describes (especially to laypeople) what’s going on, even if it may not be the most mathematically correct or rigorous word to describe what’s going on. I think you’re being needlessly pedantic.
Sounds like a very narrow view on what constitutes science. There are many fields of science where there is existing data against which new ideas can be tested without additional 'real-world' measurements. Newton's theory of gravitation relied entirely on pre-existing astronomical data for which there was no existing unifying theory. He made progress by putting forward a theory which explained that data. Now you can argue that it's not really science unless you include the original data collection and subsequent real-world measurement validation steps. But I'd be comfortable saying that Newton was indeed a scientists and did make progress in science despite only doing what some might say is the 'middle' part of the process. There are plenty of modern analogs where work like this sits out there waiting to be done using existing data.
Yes. But you don't need to monitor glucose levels to do this. Because it's well known you can productively use as much as you could manage to consume (without vomiting). So all the device will ever tell you (if you're in the middle of significant endurance event) is that you need to raise your blood sugar. It'd be no more actionable than a writing 'your blood sugar is low' on your hand with a sharpie.
If 'democratic values' are everyone having an equal voice in how the systems of a country are arranged. And if those people, through a democratic process, decide that the systems of their country shouldn't support the exploitative elements of capitalism, then it would definitely be democracy in action to make that come true. Anything counter to that would be, by definition, anti-democratic.
When you say "socialism conflicts directly with democratic values" I genuinely have no idea what "democratic values" you're talking about.
So is a zip file of source code. There is no 'compilation process'. Model training isn't a fixed end point. You start with random weights. You train. The weights get better. You do this iteratively. At some point you say 'good enough' and release. People doing further training would start with those weights and further iterate. Demanding the original training data and training process would be equivalent to demanding a completely replay-able sequence of editing steps starting with a blank text file and allowing you to replay ever edit that led the original author to the released source code.
Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).
99% of AI users these days aren't using a 'raw' LLM. They are interacting with a harness that includes tools to do search of the live (or recently crawled) web. And so the output users actually see could absolutely include correct citation of sources. So the 'architecturally impossible' bit may be technically correct but it is not practically relevant. Now whether those harnesses do a good job of orchestrating LLM output to get accurate citations (and whether they are transparent about their process) is another thing entirely. But if you're contemplating 'what LLM's can do' and aren't taking into account the harness and tooling ecosystem they are embedded in then you're missing the point.
-the thing that motivates you doesn't have to be the same thing that teaches you (e.g. a book can be a useful learning tool even if it isn't human)
-you can absolutely choose to interact with an LLM in ways that generate productive struggle. If you naively ask it for answers it won't be much use in this regard. But that's just a poor choice of how to use the tool. Just like hitting yourself on the head with a book doesn't mean the problem is with the book as learning tool.
-You don't need structured content 'on long scales' to learn. You learn a little bit at a time. Long textbooks are a side effect of the economics of moving paper around. Not a prerequisite for learning. An LLM will happily mimic the table of contents of a textbook or an outline for course. And you can use future prompts to generate the bits you need as you need them. One-shoting a textbook is a silly use of the tool. Plus, we're not short on textbooks. We're short on ways students can get customized help: answers to the specific ill-posed questions they have in their heads. LLM's are quite good at this. You can have textbooks AND LLM's (and professors)
-LLM's are happy to disagree with you. Just ask them to and they oblige. Ask them to respond as if they were a tutor and they will mimic the assuredness if appropriate.
reply