Generally Infiniband [1] or Omni-Path [2] interconnects between the compute nodes - extremely high throughput and low latency communication on the order of 100 Gb/s and 1 microsecond. These problems aren't trillions of completely independent operations which may work for cloud/distributed architecture. For these types of calculations compute node 6523 needs to know the data coming from compute node 2447 in near-realtime to continue working.

https://en.wikipedia.org/wiki/InfiniBand

https://en.wikipedia.org/wiki/Omni-Path

Thanks for the detail. I can clearly see the benefit of a couple orders of magnitude less latency.