The XUL addon system essentially allowed addons and official components alike to monkey patch any aspect implemented in XUL and indeed each other. The potential for interaction between them is therefore unlimited. When you say that its "just a matter of redesign" you are saying that if only infinite resources could be expended to avoid you never having to change anything at all the world could be a grand place. What you are failing to understand is that the browser is being secured first and foremost not against extension developers but against malicious actors. If the entire way the browser interacts is it self subject to monkey patching its infinitely harder and perhaps impossible to properly secure the single most exposed application vs the outside world.

Ideally the functionality once implemented as monkey patching can be provided by a defined interface with the addon system and indeed this is exactly what one sees today with most broadly useful extensions ported.

Incidentally if you want to analogize Firefox vs Linux the interface that is not supposed to break is the interaction with the web itself to the degree that this is possible. Linux actually makes external out of tree addons which touch let alone modify the kernel break/update themselves constantly. Linus would never have been stupid enough to commit to something like XUL for the kernel wherein he himself would be responsible for keeping thousands of monkey patching addons working while trying to evolve the kernel.

When basically whole implementation is API (which is the case for XUL), you can't really change anything to never break it. Which is what happened - Firefox stagnated, yet managed to break extensions with every minor release.