Good question. For one, there's also a custom optimized-debug backend that's used by default for Debug builds which is a lot faster than LLVM. Secondly, there's incremental compilation with an object cache so minor changes only general minor rebuilds.
For a clean release build, however, LLVM will dominate the compile time as expected. Compared to C/C++, however, there is less duplication of ODR'd code (ie- methods defined in headers such as template methods) and less debug information duplication so backend time is lower.
Since you are working on debug-specific backend, have you considered Cranelift [1]? In case you never heard of it, it was especially intended for fast codegen (because it's being used to compile WASM in Firefox), and Rust wants to be able to use it for their own debug build. If you were aware of it and decided not to use it, I'd love to know what are your rationals.
For a clean release build, however, LLVM will dominate the compile time as expected. Compared to C/C++, however, there is less duplication of ODR'd code (ie- methods defined in headers such as template methods) and less debug information duplication so backend time is lower.