> As described above, it is important to “warm-up” JavaScript when benchmarking, giving V8 a chance to optimize it. If you don’t do that, you may very well end up measuring a mixture of the performance characteristics of interpreted JS and optimized machine code.
Since unwarmed first execution is a very common use case on the web, and a very intentional improvement for WASM, it seems foolish to discard that when comparing. I understand for benchmark determinism warming is important in long-running systems, but when parsing/warming is a common part of every run, it deserves to be factored in. I can make WASM look better by removing intentional benefits from JS before comparing too.
You even included in the quote: It’s important to warm-up the code is so you don’t measure a mixture of performance characteristics between interpreted JS and compiled jS. How long the warmup takes is _incredibly_ device dependent, so instead I measured Ignition and SparkPlug independently so you get a feel for the speedup. I did not discard that at all in the comparison.
My mistake if you are actually including the warm up times in the benchmarks, I misunderstood. Arguably, single-execution-from-scratch JS vs WASM benchmarks could have even more value than repeated, warmed-up benchmarks as the former simulates a common web use case. IMO you _should_ very well end up measuring a mixture of the performance characteristics of interpreted JS and optimized machine code if that represents common use.
* You present TurboFan performance as the speed of JavaScript. e.g. your tables present JavaScript with Ignition as ~40x slower than JavaScript in certain test runs.
It's a fantastic analysis, and I am a bit surprised by some of the results. Thanks for doing the work and putting together a great resource.
kodablah's point is a valid one generally. A parallel post notes that a developer can't avoid the warm-up time, but they can by using WASM.
As an aside, does v8 cache the optimized native code to any degree? I know there is code caching in the major browsers to presumably avoid reparsing Javascript, but if I had a theoretical page with say an image blur function, would each visit/load go through the same analysis/optimization process, going from slow to fast?
I imagine such caching is slowly being phased out because it can be used to create 'super-cookies'. That is, you can fingerprint a user by detecting whether certain bits of javascript are or aren't cached. (Detection of being cached is just a matter of measuring execution time).
It _is_ cached, but the wasm binary itself as well as the optimized version to improve startup times. The cache however is per origin. So no other origin can make use of the cache which prevents the fingerprinting aspect.
As a programmer, there's a lot you can do to make your JS run faster under optimization (e.g. avoid deopt) but there's little you can do about the warm up (besides reducing binary size).
So, when you're trying to progressively improve performance of specific code (e.g. boost FPS) the warm up time is better ignored; it's not under your control.
Reducing size or switching to WASM are both things that you can do.
You should measure the cases you actually care about. For a long running app, start-up time is probably not the most important thing, for other apps its very important.
If I'm trying to improve boot time of my node app, then I must benchmark the cold behavior, (the interpreter). If I'm judging template engine performance, I probably want to do that with a hot VM, because that's going to mostly be steady state performance I'm looking at. The major exception to that may be if I have an exceedingly aggressive caching strategy, where most pages are generated shortly after a deployment (evict all the old pages with whatever the new code generates as output).
If you include "warmup" time in JS, then you must also include compile time with WASM (EDIT: to be clear, time to pre-compile wasm from generic bytecode into binary optimized for the particular architecture -- NOT time to compile whatever language into WASM in the first place).
If you're running a given piece of code only once, the interpreted code is almost guaranteed to be MUCH faster than compiling and then executing a large pile of code.
That makes little sense (assuming you mean by "compilation" the compilation to WASM from a higher-level language). The entire point is (or rather should be) to measure runtime impact on the user. If compilation (e.g. parsing, JIT, etc) is part of runtime, you measure that. I wouldn't expect a JS benchmark to measure optimizer/minifier time either.
WASM code is byte code that (after being downloaded) is pre-compiled into (presumed) safe native code to be executed. It is designed to be cross-platform and as such, isn't particularly close to any given architecture. This means that there is still room (and necessity) to optimize for a particular architecture when compiling.
Let's say you have some trivial task. You write and compile to wasm. You also write in JS. It'll be a couple kilobytes for JS and probably a couple hundred kilobytes for wasm with all your dependencies to get a usable environment.
Once downloaded, the JS and wasm bytecode must now be parsed. The JS interpreter starts parsing and running. Meanwhile the wasm code is pre-compiling into native code so it can start execution.
The tiny bit of JS takes 10ms to run in the interpreter. The larger WASM file takes 20ms to parse and 0.1ms to run. Which was faster?
That depends on how long the software runs. WASM only makes sense at the inflection point where the execution lasts long enough to counteract the parse time. This shouldn't come as a surprise. Compile lag is one reason why interpreted languages saw real-world use in the first place.
It's telling that v8 used to NOT have an interpreter, but then added one. Everything was first converted to byte code then run. This resulted in slower startup and slower overall performance. Now, the browser starts interpreting while also converting to bytecode in the background before switching over execution. Functions don't actually need a couple hundred runs in order for the optimizer to know what types they receive. Most functions could be optimized with a very high degree of success after only a couple executions. This isn't done because the time cost of optimizing would outweigh the benefit on code that isn't executed frequently.
> Once downloaded, the JS and wasm bytecode must now be parsed. The JS interpreter starts parsing and running. Meanwhile the wasm code is pre-compiling into native code so it can start execution.
I thought wasm's format was specifically designed so that parsing and compiling could be performed in a streaming fashion, so that you don't have to wait for the download to finish.
JS can already be parsed as it is streamed (I think this has been the case since around Chrome v40 and earlier for Firefox). The JS binary AST proposal could make that even more efficient.
My example assumed you were running the code locally. If it's coming over the wire, larger WASM binaries will suffer an additional penalty over the network because download speed is much slower than parse speed.
Since unwarmed first execution is a very common use case on the web, and a very intentional improvement for WASM, it seems foolish to discard that when comparing. I understand for benchmark determinism warming is important in long-running systems, but when parsing/warming is a common part of every run, it deserves to be factored in. I can make WASM look better by removing intentional benefits from JS before comparing too.