Headline marketing "AI TOPS" is also based on whatever smaller format a new generation supports. E.g. doubling headline marketing performance from Ada to Blackwell by going from FP8 to FP4.
Don't forget "structured sparsity" where two out of every four weights must be zero. Another trick that Nvidia still happily assume for the sake of a graph or an headline figure.
