I don't trust the Java sandbox to protect me while running arbitrary code. So I don't use the Java browser plugin.
But I don't see how the desktop runtime is a security hole (you're running binaries you trust outside a sandbox) and I very much doubt it's any more buggy than your average Python, Ruby, Node.js, etc. runtime. Or, for that matter, your average native library.
Can you (easily) install one without the other? I know I can go around my browsers and disable the plugins after an install, but then I'm going to have to do the same thing each time there's a java security update and the runtime gets upgraded. Not to mention Oracle's crappy habits of bundling toolbars & other crud along with each security patch.
But I don't see how the desktop runtime is a security hole (you're running binaries you trust outside a sandbox) and I very much doubt it's any more buggy than your average Python, Ruby, Node.js, etc. runtime. Or, for that matter, your average native library.