Figuring out which of two things is bigger is hard. The article lightly touches on the two main reasons:
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.
Isn't this why benchmarking is used? You select some common payload and then use that as a realworld simulation of how your architecture performs with it.
Of course this comes with its own can of worms, like overfitting and such, but I could imagine a benchmarking solution that gives you a more granular look at which specific tasks an architecture performs well.
For the specific purpose of what the article presents, I feel something simpler like 10%/50%/90% times can be better (two systems that have times as 0.1/1/10 or 0.9/1/1.1 will have "the same average" but where one might be adequate the other one might not)
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.