Is it a coincidence that USA and Canada now have the same amount, but they equalised without transferring any to Canada, and they can say they reduced in both?
Yes, I agree that this is the right solution, because for a locally-hosted model I value more the quality of the output than the speed with which it is produced, so I prefer the models as they were originally trained, not with further quantizations.
While that paper praises the Apple advantage in SSD speed, which allows a decent performance for inference with huge models, nowadays SSD speeds equal or greater than that can be achieved in any desktop PC that has dual PCIe 5.0 SSDs, or even one PCIe 5.0 and one PCIe 4.0 SSDs.
Because I had also independently reached this conclusion, like I presume many others, I have just started to work a week ago on modifying llama.cpp to use in an optimal manner weights stored on SSDs, while also batching many tasks, so that they will share each pass through the SSDs. I assume that in the following months we will see more projects in this direction, so the local hosting of very large models will become easier and more widespread, allowing the avoidance of the high risks associated with external providers, like the recent enshittification of Claude Code.
> While that paper praises the Apple advantage in SSD speed, which allows a decent performance for inference with huge models, nowadays SSD speeds equal or greater than that can be achieved in any desktop PC that has dual PCIe 5.0 SSDs, or even one PCIe 5.0 and one PCIe 4.0 SSDs.
Apple’s advantage is their unified memory architecture where the CPU, GPU and Neural Engine share the same memory and the SSD is directly connected to the SoC--less latency than PCIe. Memory bandwidth starts at 300+ GB/s.
In an optimized implementation of model inference, the latency of SSD access has no importance, because no random accesses are done.
The purpose of optimizing model inference for weights stored on SSDs is to achieve a continuous reading from SSDs at the maximum throughput provided by hardware, taking care that any computations and any accesses to the main memory are overlapped over the SSDs reading.
Many of the keyboard commands are configurable in settings, complete with switching cmd and ctrl keys around.
Or you can get used in a week or two when switching, this is what I did years ago and now for me Win/Linux is confusing and find the location of the command key more ergonomic on a Mac.
A18 Pro is generations ahead of M1 and M2 on single thread if these scores are true.
Are you saying we had this incredibly overpowered silicon shipped on millions of Instagram machines?
I'm physically hurting at the amount of processing power we wasted. Atleast Apple did the right thing here.
Mobile devices, to give you good battery life, operate in a “race to sleep” mode. Good performance is necessary not to compute a lot of things, but to finish your computation quickly and shut down the power hungry processors. So do not worry, the performance is not being left on the table. It is there so that your phone lasts throughout the day.
A series chips also use a slightly different instruction set with memory tagging and pointer authentication. IIUC it's Armv9, which the M series chips don't support.
No, he shared on Dithering that there was a back-and-forth email exchange about the execs appearing on the Talk Show as usual, but they couldn't come to an agreement on details of the event.
reply