However, normal distributions actually behave benignly and are a prerequisite for some statistical tests.
This is where my first comment gets in, that for my benchmarking statistical tests are not necessary at all ...
---
The author writes that he only measures minimum and average, but the maximum values are of course also important and worth to measure.
The author's key point is that without a normal distribution, the maximum values can be highly scattered. I agree and basically it is always nice to know the distribution of the random variable.
The article questions the normal distribution assumption for benchmark timings.
It says that this assumption would be important for the "standard error of the mean":
"Your error should go down as the square root of the number of measures."
Standard distribution is not even necessary for this:
https://en.wikipedia.org/wiki/Standard_error
However, normal distributions actually behave benignly and are a prerequisite for some statistical tests.
This is where my first comment gets in, that for my benchmarking statistical tests are not necessary at all ...
---
The author writes that he only measures minimum and average, but the maximum values are of course also important and worth to measure.
The author's key point is that without a normal distribution, the maximum values can be highly scattered. I agree and basically it is always nice to know the distribution of the random variable.