arXiv link for the paper: https://arxiv.org/pdf/2506.08257

(Note that the author Lukas Beyer is distinct from the other well-known author in the field, Lucas Beyer)

Their sketchy ad/malware era aside, I do appreciate sourceforge keeping all the old OSS websites and repos alive.

If you find this kind of stuff interesting, I'd strongly recommend checking out the book "Spark: The Life of Electricity and the Electricity of Life" by Jorgensen

Looking forward to Casey Muratori's talk!

When I saw the title of the conference I immediately thought of him so I'm not surprised he's headlining!

I wonder if this is a result of the previously reported clashes between OpenAI and Microsoft over access to the Windsurf IP (under their investment agreement)

The open training data is a huge differentiator. Is this the first truly open dataset of this scale? Prior efforts like The Pile were valuable, but had limitations. Curious to see how reproducible the training is.

> The model will be fully open: source code and weights will be publicly available, and the training data will be transparent and reproducible

This leads me to believe that the training data won’t be made publicly available in full, but merely be “reproducible”. This might mean that they’ll provide references like a list of URLs of the pages they trained on, but not their contents.

Well, when the actual content is 100s of terabytes big, providing URLs may be more practical for them and for others.

The difference between content they are allowed to train on vs. being allowed to distribute copies of is likely at least as relevant.

No problem, we have 25 Gbit/s home internet here. [1]

[1] https://www.init7.net/en/internet/fiber7/

That wouldn't seem reproducible if the content at those URLs changes. (Er, unless it was all web.archive.org URLs or something.)

This is a problem with the Web. It should be easier to download content like it was updating a git Repo.

Yeah, I suspect you're right. Still, even a list of URLs for a frontier model (assuming it does turn out to be of that level) would be welcome over the current situation.

Yup, it’s not a dataset packaged like you hope for here, as it still contains traditionally copyrighted material