Stupid question from me: how the hell do they prototype and debug this? It's very far from my EE knowledge. Communication analysers will show the eye diagram and tell you a lot about stray capacitance and inductance, but I know that most 10GSa/s+ devices cost more than a house and have a tendency to live in a lab for a long time. It seems to my mind like there are so many difficulties in driving pci-e fast, from timing to voltage drop and current limitations. All of this must be implemented, somehow, if the spec is finalised. Right? They wouldn't write it up otherwise (right?!).
So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?
Any communication protocol has functional layers to them, and at the very bottom is what's called the Phy layer (or physical layer.) While pretty much every layer can be implemented in digital logic, the physical layer has a lot of analog circuitry. (It's programmable analog circuitry, so it's technically mixed signal.) Most wired communication protocols that you are familiar with (including SAS/SATA, USB, HDMI/DP, PCIe/NVMe, and Ethernet) all use what's known as a "SerDes Phy" which is short for Serializer Deserializer. It takes parallel data in, and serializes it to transmit data, and does the reverse when receiving data. In the example of PCIe Gen 4, it takes 32 bits of data in at 500 Mb/s (per bit) and serializes it to 16 Gb/s.
Because the SerDes Phy is such a common part of all communication protocols, it's often reused in other designs. For example - a GPU may use the same SerDes Phy design for its PCIe lanes as it does for its HDMI ports (at least on the transmit side.) They are however programmed differently.
Because these designs see so much reuse in so many different chips, there are companies that specialize in designing SerDes Phys, and their development is usually ahead of protocol development. Though in today's day and age, it's super complicated, and consists of way more than shift registers and a PLL. But there are SerDes designs that exist in products that are capable of 112Gb/s (not sure if 224Gb/s exists outside of test chips yet.)
You can buy modern FPGAs that have PCIe Gen 5, but with a SerDes Phy capable of going way faster. And if you want, you could bypass that hard block and talk straight to the SerDes, and program your own PCIe 6.0 controller.
But to answer your question on how they prototype it and test it, they run computer simulations that are exceedingly complicated and slow, and once a test chip comes back from the fab, eye diagrams are usually collected using what's known as a repeating signal scope. Since it's insanely difficult to sample a 112Gb/s signal at any reasonable resolution, it's sampled statistically (in time) over billions of samples, and plotted by knowing the frequency of the target signal. An eye diagram is what comes out of a traditional NRZ (non return to zero) where each cycle represents either a 1 or a 0. In an eye diagram, you see all permutations of bit transitions overlayed on each other, since you're sampling billions of cycles statistically. This is important because you can see other anomalies of your signal like jitter and rise/fall times.
There's a lot more to it, and this comment is already getting to be a wall of text but definitely feel free to ask more questions!
Do you happen to know why 10Gb Ethernet is still so expensive when PCIe and USB involve much higher bitrate PHYs by the dozen? Is it just a business thing? Or do 10GbE PHYs do something funky that has prevented them from sliding down the price curve, even after all this time?
You can get 10 Gb networking for fairly affordable prices with say, Mikrotik hardware.
I think the reason why it costs a lot is that packets are still tiny, and ports are many. A quality 16 port, 10 Gb switch is expected to have no internal bottleneck, able to handle every port being used to full capacity at once. And that works out to 320 Gbps bandwidth in the whole switch.
Then there's that thanks to the Internet, we're still stuck with 1500 byte packets, which means best case we're dealing with 812,744 pps, and worst case, we're dealing with 14,880,952 pps. And that's on one port.
With the later number, your time budget per packet is 67ns. And if you start looking, context switches, cache misses, locks, etc all have times also measured in nanoseconds.
It's getting very, very tight and requires extremely careful programming to actually make this work. And of course in modern times, we have firewalls, NAT, statistics, routing, etc all of which cost precious time out of this budget, and which are all user-configurable, so it's something that's difficult to just shove into a purpose-built chip to take that load off the CPU.
PCIe has the advantage of that it doesn't have to deal with half the crazy stuff networking involves.
PCIe is a packet switched network with point-to-point routing, firewalls, and link aggregation. Everything is done on dedicated silicon because it simply can't be done in software -- forget having a cache miss, the PCIe bus itself might be the mechanism to serve the cache miss. Or ship a bunch of 400Gbps ethernet traffic... speaking of which, picking a bandwidth fight with PCI express is probably not the best move ever :)
> 16 port, 10 Gb ... 320 Gbps bandwidth in the whole switch
That's a nice switch. Expensive, but nice. Let's do the math on a modern low-end consumer CPU:
PCIe: 20x 32Gb/s x2 = 1280Gbps
Chipset: 8x 16Gb/s x2 = 256Gbps
Display: 4x 4x 12Gb/s = 192Gbps
...and if we compared the switch to a CPU in its own price bracket, this would go from silly to ludicrous.
Look, I'm not asking for a battery of 400GbE ports on every laptop, but USB can deliver 10Gb/s across a few meters for a few dollars while Ethernet wants a few hundred dollars to ship the same bits over the same distance, and that's unfortunate.
People keep trying to make excuses for Ethernet, but I don't think it deserves their efforts. I think something's broken in the consumer networking industry -- it feels like they gave up on consumers entirely after 1GbE, and I think we could all benefit if someone breathed some life back into that corner of the market.
Distance and signal integrity would be my guess. The longer your transmit the signal, the more complicated and expensive it is to maintain signal integrity. There are out of spec USB cables that manage to get longer distance by converting to fiber optic (meaning it’s an active cable) and even then it’s extremely expensive for not that much cable.
Intense preemphasis and equalization are table stakes for high speed serial these days. Have been since PCIe3. By Gen5, the fundamental lobe goes to 32GHz, and the mandates are steep. It has to be able to fish the signal up after the dielectrics have eaten all but a few parts per thousand of the high frequency energy! Compared to modern transceivers -- these new ones or even, if you'll indulge my speculation, the transceivers in your own computer that are going unused by the dozen -- compared to those, dual simplex 10Gb/s just looks... old. The price doesn't really match up, and I wonder if it's a business thing or if there is an unfortunate design choice that baked in some manufacturing difficulty.
I don’t know anything about what you just said as I don’t have that level of digital/analog design knowledge to know how to respond. All I know is that USB 3.0 cable lengths is 18 meters but to get there the cable becomes incredibly thick, heavy and inflexible. ePCIe cables are measured in cm. Ethernet can go up to 100m at 10GBe or more if you’re willing to sacrifice some speed (or faster if you sacrifice some cable length)
Could switching to PCIe instead of Ethernet for the digital signaling have better performance? I don’t know but I doubt it. Ethernet is hitting 100gbps and higher and I suspect the challenge is designing cabling that can hit just as much as the analog design piece. Not to mention that USB3.2 and Ethernet have very similar speeds, just Ethernet can manage it over much longer distances at significantly lower cost. I’m skeptical PCIe has some kind of magic bullet here. These protocols are optimized for totally different use cases.
As someone without a background in EE I find it fascinating how we are able to build hardware that operates and deals with data at high speeds that it needs to be debugged using statistical analysis.
How is it even possible to have correctly functioning circuits at such high speeds? I'm guessing at the very bottom of the Phy layer there several transistors switching at incredibly fast speeds. I understand that these transistors have several parameters such as their delay, rise, and fall time. Things most certainly don't happen instantaneously, a few logic ICs I played with also have a propagation delay time specified in their data sheets lasting a few nanoseconds.
Take the deserializer for an example, happening at 16Gb/s, which if I understand correctly equates to 1 bit every 62.5ps (0.0625ns). How does any circuit affected by the parameters I just talked about handle signals at such high speeds?
(Please go easy on me, I know some of my assumptions are probably wrong or oversimplified but I would love to learn more)
Select 30nm CMOS processes already had fT (frequency at which transistor no longer has gain) of 300GHz, so the component-level limits are probably flirting with THz by now. Of course, you need to knock off an order of magnitude to build useful comparators/etc out of those, and that gives you... 100GHz, right about what we see on those high speed SerDes :)
One more thing: CMOS historically wasn't the fastest logic family. Bipolar ECL was faster, but it burned a ton of idle current, so it ran into power limitation long before CMOS.
In either case, thermal limits prevent you from using these speeds across the whole chip, but just because your thermal budget can't afford 10,000,000,000 speed demons doesn't mean it can't afford 1000 of them in a few SerDes :)
But yeah, decades of Moore's law have not gone to waste, and those discrete logic chips are frozen in time. It's a neat reminder of how far we have come!
Is it the SerDes component that also does the word encoding (8b10, etc., forgot the proper terminology for this) ?
I work on the Hardware/Software interface quite a bit (more in the past than now) and this is always fascinating, I'm on the Software side, thanks for the details!
The encoding takes 8 bits and turns them into 10 for DC balance and to help the clock recovery system. You are basically losing 20% of your bandwidth to encoding overhead.
PCIE Gen 3 and above have switched to 64b/66b encoding for less overhead.
At a higher level, considering that all protocols are serial these days, how does a multi-lane protocol (PCIe x4 or x16, Ethernet x4) use the multiple lanes, is it multiple independent requests going on each lane?
PCIe "shreds" each packet across multiple lanes to reduce latency.
Generally, if you are wondering what PCIe does, the answer is usually "they do the hardest, fastest, highest-performance thing and they do it so well that nobody notices."
I'm not in EE, but I'd imagine that you start with x1 lane and work your way up.
Each PCIe lane is an independent SERDES. Its not like a parallel memory controller where they share a clock, when you have x16 lanes, you have 16x independent streams you're shoving to the device downstream.
We build a test chip with the serdes and some control logic. One wafer might give us 50 chips. We package them and build a board with two of the chips and have them talk to each other.
PCI SIG has regular validation events and beyond that the lab where I cut my teeth is well renowned for hosting plugfests for cutting edge technologies.
Yep, and the highest data rates are only really possible on the best processes (12nm, 7nm, 5nm) , which is a fun chunk of change to spend for a tape out.
Modern high end serdes have very complex signal mgmt and monitoring tools built in. You can detect the link quality, pull eye diagrams/etc from them.
So a lot of the reason for using some of these high end scopes/analyzers is bypassed by the testing/diagnostics functions on the serdes itself. And then of course there are a ton of signal tuning parameters which get programmed during link training.
Even modern analyzers have problems with this, and I've seen vendors that instead of even trying to reconstruct the signal at the endpoint use virtual transmission models and require one to probe at the source.
This is one of those areas that I bump into every couple years, and its frequently quite educational.
I suspect it's heavily sampled, and maybe not only on the time axis. They would already have a reconstructed clock available, so it would be straightforward to sync up a time aperture, and the voltage aperture could be swept by adding an offset into the output comparator(s). They could then scan the ranges of both parameters to build up the eye diagram with minimal additional hardware. This would make it a brilliant oscilloscope if you want an eye diagram of a high speed serial bus, but for just about any other task it would be a bit rubbish. There isn't a UXR hiding in every PCIe 5+ lane.
Haha, yes, I'd love to see what slathering vaseline on a high speed diff pair does to the eye. Link equalization is the real MVP! It'll be fun to see how this evolves with 32Gbps and PAM4.
You better make isolation a subpart of 'some signal conditioning' or you'll be looking at some expensive repairs. That's the hard part of making a scope: isolating the source and the scope guts while maintaining signal integrity.
So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?