That would rival the speed of on-die processor caches. Right now the Apple M3 offers 300 GB/s or 2.4 Terabits / sec.

A 1 terabit /sec network connection would allow you to basically allow your machine to train and tune arbitrarily sized models completely remotely.