It's possible you're underestimating the open source community.
If there's a competing platform that hobbyists can tinker with, the ecosystem can improve quite rapidly, especially when the competing platform is completely closed and hobbyists basically are locked out and have no alternative.
> It's possible you're underestimating the open source community.
On the contrary. You really don't know how I love and prefer open source and love a more leveling playing field.
> If there's a competing platform that hobbyists can tinker with...
AMD's cards are better from hardware and software architecture standpoint, but the performance is not there yet. Plus, ROCm libraries are not that mature, but they're getting there. Developing high performance, high quality code is deceivingly expensive, because it's very heavy in theory, and you fly very close to the metal. I did that in my Ph.D., so I know what it entails. So it requires more than a couple (hundred) hobbyists to pull off (see the development of Eigen linear algebra library, or any high end math library).
Some big guns are pouring money into AMD to implement good ROCm libraries, and it started paying off (Debian has a ton of ROCm packages now, too). However, you need to be able to pull it off in the datacenter to be able to pull it off on the desktop.
AMD also needs to be able to enable ROCm on desktop properly, so people can start hacking it at home.
> especially when the competing platform is completely closed...
NVIDIA gives a lot of support to universities, researchers and institutions who play with their cards. Big cards may not be free, but know-how, support and first steps are always within reach. Plus, their researchers dogfood their own cards, and write papers with them.
So, as long as papers got published, researchers do their research, and something got invented, many people don't care about how open source the ecosystem is. This upsets me a ton, but when closed source AI companies and researchers who forget to add crucial details to their papers so what they did can't be reproduced don't care about open source, because they think like NVIDIA. "My research, my secrets, my fame, my money".
It's not about sharing. It's about winning, and it's ugly in some aspects.
That said, for hobbyist inference on large pretrained models, I think there is an interesting set of possibilities here: maybe a number of operations aren't optimized, and it takes 10x as long to load the model into memory... but all that might not matter if AMD were to be the first to market for 128GB+ VRAM cards that are the only things that can run next-generation open-weight models in a desktop environment, particularly those generating video and images. The hobbyists don't need to optimize all the linear algebra operations that researchers need to be able to experiment with when training; they just need to implement the ones used by the open-weight models.
But of course this is all just wishful thinking, because as others have pointed out, any developments in this direction would require a level of foresight that AMD simply hasn't historically shown.
IDK, I found a post that's 2 years old that has links to doing llama and SD on an Arc [0] (although might be linux only), I feel like a cheap huge ram card would create a 'critical mass' as far as being able to start optimizing, and then from a longer term Intel could promise and deliver on 'scale up' improvements.
It would be a huge shift for them. To go from preferring some (sometimes not quite reached) metric, to, perhaps rightly play the 'reformed underdog'. Commoditize Big-Memory ML Capable GPUs, even if they aren't quite as competitive as the top players at first.
Will the other players respond? Yes. But ruin their margin. I know that sounds cutthroat[1] but hey I'm trying to hypothetically sell this to whomever is taking the reigns after Pat G.
> NVIDIA gives a lot of support to universities, researchers and institutions who play with their cards. Big cards may not be free, but know-how, support and first steps are always within reach. Plus, their researchers dogfood their own cards, and write papers with them.
Ideally they need to do that too. Ideally they have some 'high powered' prototypes (e.x. lets say they decide a 2-gpu per card design with an interlink is feasible for some reason) to share as well. This may not be be entirely ethical[1] in this example of how a corp could play it out, again it's a thought experiment since intel has NOT announced or hinted at a larger memory card anyway.
> AMD also needs to be able to enable ROCm on desktop properly, so people can start hacking it at home
AMD's driver story has always been a hot mess, My desktop won't behave with both my onboard video and 4060 enabled, every AMD card I've had winds up with some weird firmware quirk one way or another... I guess I'm saying their general level of driver quality doesn't lend to hope they'll fix dev tools that soon...
ROCm doesn't really matter when the hardware is almost the same as Nvidia cards. AMD is not selling "cheaper" card with a lot of RAM, what the original poster was asking. (and a reason why people who like to tinker with large model are using Macs).
You're writing as if AMD cares about open source. If they would only actually open source their driver the community would have made their cards better than nvidia ones long ago.
I'm one of those academics. You've got it all wrong. So many people care about open source. So many people carefully release their code and make everything reproducible.
We desperately just want AMD to open up. They just refuse. There's nothing secret going on and there's no conspiracy. There's just a company that for some inexplicable reason doesn't want to make boatloads of money for free.
AMD is the worst possible situation. They're hostile to us and they refuse to invest to make their stuff work.
> If they would only actually open source their driver the community would have made their cards better than nvidia ones long ago.
Software wise, maybe. But you can't change AMD's hardware with a magic wand, and that's where a lot of CUDA's optimizations come from. AMD's GPU architecture is optimized for raster compute, and it's been that way for decades.
I can assure you that AMD does not have a magic button to press that would make their systems competitive for AI. If that was possible it would have been done years ago, with or without their consent. The problem is deeper and extends to design decisions and disagreement over the complexity of GPU designs. If you compare AMD's cards to Nvidia on "fair ground" (eg. no CUDA, only OpenCL) the GPGPU performance still leans in Nvidia's favor.
That would require competently produced documentation. Intel can't do that for any of their side projects because their MBAs don't get a bonus if the tech writers are treated as a valuable asset.
If there's a competing platform that hobbyists can tinker with, the ecosystem can improve quite rapidly, especially when the competing platform is completely closed and hobbyists basically are locked out and have no alternative.