Even if LLMs didn't advance at all from this point onward, there's still loads of productive work that could be optimized / fully automated by them, at no worse output quality than the low-skilled humans we're currently throwing at that work.
inference requires a fraction of the power that training does. According to the Villalobos paper, the median date is 2028. At some point we won't be training bigger and bigger models every month. We will run out of additional material to train on, things will continue commodifying, and then the amount of training happening will significantly decrease unless new avenues open for new types of models. But our current LLMs are much more compute-intensive than any other type of generative or task-specific model
Run out of training data? They’re going to put these things in humanoids (they are weirdly cheap now) and record high resolution video and other sensor data of real world tasks and train huge multimodal Vision Language Action models etc.
The world is more than just text. We can never run out of pixels if we point cameras at the real world and move them around.
I work in robotics and I don’t think people talking about this stuff appreciate that text and internet pictures is just the beginning. Robotics is poised to generate and consume TONS of data from the real world, not just the internet.
While we may run out of human written text of value, we won't run out of symbolic sequences of tokens: we can trivially start with axioms and do random forward chaining (or random backward chaining from postulates), and then train models on 2-step, 4-step, 8-step, ... correct forward or backward chains.
Nobody talks about it, but ultimately the strongest driver for terrascale compute will be for mathematical breakthroughs in crypography (not bruteforcing keys, but bruteforcing mathematical reasoning).
Yeah, another source of "unlimited data" is genetics. The human reference genome is about 6.5 GB, but these days, they're moving to pangenomes, wanting to map out not just the genome of one reference individual, but all the genetic variation in a clade. Depending on how ambitious they are about that "all", they can be humongous. And unlike say video data, this is arguably a language. We're completely swimming in unmapped, uninterpreted language data.
Inference leans heavily on GPU RAM and RAM bandwidth for the decode phase where an increasingly greater amount of time is being spent as people find better ways to leverage inference. So NVIDIA users are currently arguably going to demand a different product mix when the market shifts away from the current training-friendly products. I suspect there will be more than enough demand for inference that whatever power we release from a relative slackening of training demand will be more than made up and then some by power demand to drive a large inference market.
It isn’t the panacea some make it out to be, but there is obvious utility here to sell. The real argument is shifting towards the pricing.
> We will run out of additional material to train on
This sounds a bit silly. More training will generally result in better modeling, even for a fixed amount of genuine original data. At current model sizes, it's essentially impossible to overfit to the training data so there's no reason why we should just "stop".
You'd be surprised how quickly improvement of autoregressive language models levels off with epoch count (though, admittedly, one epoch is a LOT). Diffusion language models otoh indeed keep profiting for much longer, fwiw.
"On the other hand, training on synthetic data has shown much promise in domains where model outputs are relatively easy to verify, such as mathematics, programming, and games (Yang et al., 2023; Liu et al., 2023; Haluptzok et al., 2023)."
With the caveat that translating this success outside of these domains is hit-or-miss:
"What is less clear is whether the usefulness of synthetic data will generalize to domains where output verification is more challenging, such as natural language."
The main bottleneck for this area of the woods will be (X := how many additional domains can be made easily verifiable). So long as (the rate of X) >> (training absorption rate), the road can be extended for a while longer.
How much of the current usage is productive work that's worth paying for vs personal usage / spam that would just drop off after usage charges come in? I imagine flooding youtube and instagram with slop videos would reduce if users had to pay fair prices to use the models.
The companies might also downgrade the quality of the models to make it more viable to provide as an ad supported service which would again reduce utilisation.
For any "click here and type into a box" job for which you'd hire a low-skilled worker and give them an SOP to follow, you can have an LLM-ish tool do it.
And probably for the slightly more skilled email jobs that have infiltrated nearly all companies too.
Is that productive work? Well if people are getting paid, often a multiple of minimum wage, then it's productive-seeming enough.
Every few years I try Wine for whatever app or game and it spits out some obscure error messages I have to Google, and the suggestion is to recompile this or that, after downloading some DLLs from a random Russian honeypot on Yandex.
As a Russian I do feel a little bit of pride whenever people complain about something like this. Not hugely proud because you're using it in a negative light, but still proud nevertheless :)
Not all functionality is perfectly replicated, so the user experience depends on the application being used.
I use it semi-regularly for quite a number of years and (consequently) various versions of Wine to run (a near current version of) LTspice. That works perfectly as far as I can tell, but it is my understanding that the maintainer of LTspice puts in some effort to assure compatibility with the then current version of Wine.
Countries like to do deals with the world's biggest economy / greatest innovators. They'll come at it more suspiciously than ever before, but America is, for now, "so good you can't ignore it"
Linus was a Finn and Tim was a Brit. So Europe can claim those two innovators - their work's turned out to be pretty useful. Don't know where you can stuff Elon, but he's mostly non-US as I recall (3 citizenships?). I'm too lazy to look up whether any of the Bell Lab people weren't natives, and I think the MIT hackers were US-born. But, perhaps, innovation moved worldwide in the 80's or 90's?
Many of the people leading the Manhattan project were European. America's greatest strength has always been its ability to attract and give opportunity to the smartest people from across the world. That is now ending.
I pulled the latest overstay data from the CBP website (2024) and compared it to the list of countries. Some of the countries have high overstay rates (Haiti and Laos >24%), but others don't. Barbados (0.44%) has a lower overstay rate than France (0.48%). Libya (1.59%) has a lower rate than Portugal (1.68%). Some countries with high rates aren't on the list entirely, like Malawi (22.05%). Also, the hypothesis fails a chi square test. It's not that.
I regularly interact with devs and project managers in Eastern Europe. Their quality is top-notch and their English is good enough you'll forget they're not natives. Most importantly, their mentality is American. Like... weirdly American.
Yes means yes, no means no, "how was your weekend" and then down to business. It's a pleasure interacting with them.
Funny story, the lead on our Eastern European team told me a while back that he had to tell his team:
When the North Americans ask at the beginning of a meeting "How's it going?", they do NOT really want to know how you are doing. It's just social lubrication before getting to work.
Before that, we were getting to learn that their mother in-laws in town or different medical issues.
reply