I think in both cases, there's a similar switch of philosophy. We went from "there are a bunch of components glued together, and with enough glue you can solve anything (except an excess of glue)" (shell scripts for sysvinit vs declarative unit files for systemd; X11 apps that can do almost anything, vs the privileged compositor for wayland), to a model of "we wanted things to just work without having to install glue, so we integrated more functionality people wanted, if you want to swap it out you have to swap out the whole thing and keep up enough with all the other features people want" (e.g. you have to actually get the things you want into unit file directives, or get the features you want into the compositors people use; or you have to build a completely separate compositor or a separate init).
That's a tradeoff. For my part, I'm thrilled that so many more things just work out of the box; however, it's discouraging for people whose features aren't covered yet, since they have to go work on integration rather than writing a specialized tool and encouraging people to glue that tool in. But the benefit of that is that once something is integrated, it just works, with no glue required.
You are presenting this in certain way that I think veers into inaccurate and misleading in an attempt to smooth things over and be nice.
What we had before systemd was -
90% glue code, reimplemented quite badly across X distributions.
That glue code was, in practice, extremely brittle and very very unfun to attempt to keep even simple daemons running "portably" distribution to distribution.
The other 10% was increasingly aging and ill maintained c code snippets.
That was not a nice world for people actually using it.
For people making stuff up about "the old days" that didn't actually participate in the misery of making basic systemv scripts, yeah it was composable and we lost something.
The internals of systemd are just as brittle, and the model of unit file configuration does not really apply cleanly beyond the simplest cases. So editing unit files becomes an undocumented dark art.
Chiming in here to say that systemd seems to cover the 80% (or even 90%) case pretty nicely, however that last 10-20% is now really difficult.
Anyone who has really delved into systemd knows this but they get shouted down as halting progress and hugging bash scripts, which is disingenuous as bash scripts (as per sysvinit) were painful and had great difficulties in areas such as determinism and parallel execution.
If you ever want an example of what I mean: look at how systemd starts mysql. Someone (not me) spent at least a man month making that work.
I do begrudge the all or nothing approach that systemd is taking (even if it claims to be modular), but I will admit openly: that 80% case is a lot nicer.
> If you ever want an example of what I mean: look at how systemd starts mysql. Someone (not me) spent at least a man month making that work.
I was curious, so I cracked open the mariadb.service unit that ships with Arch Linux. Other distros might ship different unit definitions, but this is the one I'm looking at.
It's large, yes, but very well commented and seems quite clear to me.
Much of the complexity seems to be around sandboxing: PrivateNetwork, CapabilityBoundingSet, PrivateDevices, ReadWritePaths, ProtectHome, PrivateTmp. These are all settings to do with hardening the service. They're totally optional, and can be removed without impacting functionality.
There is some extra complexity in the ExecStartPre and ExecStartPost commands. This appears to be something to do with the Galera cluster functionality. I'm not entirely sure what's happening with those, but I imagine these commands would also be present in a SysV init script implementing the same functionality.
The rest is pretty standard stuff: the user/group is set, along with the umask and some ulimits. LD_PRELOAD is set to load jemalloc. There's also some start/stop timeouts configured, with a comment explaining that these same timeout values were used in the SysV init scripts in the past.
Essentially, I'm not really sold that any of this complexity is caused by systemd. The hardening strikes me as a little unusual, and at a guess I'd say that this probably wouldn't be present in a SysV init script. If it were, the configuration would live in executable code, not in strictly declarative config directives.
> The internals of systemd are just as brittle, and the model of unit file configuration does not really apply cleanly beyond the simplest cases. So editing unit files becomes an undocumented dark art.
I call bullshit on that.
We removed few thousand lines of SysV init scripts from our configuration management that were basically fixed by us for subtle errors when we migrated to systemd. There is very little cases that aren't handled by very simple systemd units, in fact I'd dare you to give example that would be easy in SysV and hard in systemd.
Few fun cases:
Process after doing /etc/init.d/servicename start -> status immediately returned service stopped. That confused service managers like Pacemaker that thought the service failed to start. Why?
The app did start -> save pid file. Java app so it took a second.
The script just forked the app into background. So if you run status after start the pidfile was not there and it showed it is stopped. Not the problem in systemd
Another
(IIRC) MySQL init script put pid file in /var/run/, like everything else. Bit old install so /var/run wasn't a separate partition or tmpfs.
MySQL init script also didn't try to start if it found the PID existing in system. It didn't check what* was running tho.
So in crash scenario, server started, MySQL init script went "oh, there is apache daemon using that PID I had last reboot, clearly that means mysql is working" and exited. MySQL status returned MySQL working. Not a problem in systemd
Another:
Script just... sent signal and exited on stop. App could took some minutes to shut down. stop -> start failed, pid was lost coz script removed file with it after sending signal.... similar problem with multi-process app scripts not killing all childs. Systemd "just have a cgroup and mark service as stopped once everything dies" fixes that. YOu can ask for that behaviour if you tell systemd to not kill processes on stop but that's pretty much "purposefully using non-default settings" so can't be really done on accident.
IIRC all or most of that is how SysV init script by standard should not work but were simply bugged
Those scripts were in popular packages in "enterprise" linux distros. If even those maintainers can't make "simple SysV script" then maybe sysv scripts aren't as "simple" as some people claim they are.
X11 offers a lot of functionality on the server - a lot of it unused, but still. Wayland technically specifies little but the protocol. So here the philosophy switch is the other way around.
Simply the fact that systemd turned logs into a binary format that can't be read with standard tools, and this was not (easily? at all?) possible ti cgange made me really strongly dislike it.
systemd has its own tool to read its binary log format, but I've already seen it corrupt its own logs and fail to read it.
And did they do the binary format for efficiency? Get this: I've never seen anything be inefficient due to logging before systemd. Shortly after archlinux switched to it, something was being super slow. Sure enough, it was systemd not being able to handle the amount of ligs something produced.
I think systemd is very opinionated, and something that opinionated should not be such basic piece of the linux landscape. There should be choices of individual components.
I like some parts systemd does but binary format is travesty. And I do not mean "the fact it is binary is bad" but format itself. Example:
strace systemctl status haproxy 2>&1 |grep /var/log/journal |wc -l
356
356 files opened (whole logging dir is around 800MB) only to tell me
CGroup: /system.slice/haproxy.service
├─2428 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─2434 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
Warning: journal has been rotated since unit was started, output may be incomplete.
Yes, to tell me there are no logs for the app. And it takes multiple seconds (I tested it on NAS with spinning rust).
The binary log format has a huge advantage: it allows for arbitrary amounts of fields containing arbitrary content. Unlike syslog, journald is trivial to parse unambiguously, and you can even dump binary data like crash logs into it if you really want to. It also parses universally -- you always know what the timestamp is, you don't need to craft a per-service regex.
I'm not sure what would be a better alternative. I guess it could have gone with dumping JSON, but then you have issues with newlines, or escaping, and a single bad character can break parsing. At that point you might as well go binary, IMO.
If an application provides some standard command line tool that allows its internal data to be output as formatted text, then it in fact can be read with standard tools and manipulated with shell pipelines and scripts.
Plain text files are nice where possible, but binary formats are not automatically going against the Unix Way.
A well-designed, well-implemented binary log format? Quite possibly so.
Let's create a logging daemon that creates Apache Parquet files, for instance. Or at least uses a well-established, well-tested binary row format, with existing tools capable of easily unpacking it and working with it.
Maybe journald has a good enough API to stick to it. I hope it's going to be replaced with a saner implementation of that API, like Pipewire did with Pulse Audio.
redhat seem to have tried their best to use systemd to turn Linux into Windows NT
- over-complicated service manager
- binary logs that are difficult to find
- incomprehensible task scheduler
- hidden caching dns resolution service
- disk manager
- network manager
- login management
- crappy ntp client
all it needs is svchost.exe
something that was 100% reliable is now about 98% reliable, and when it inevitably breaks it's completely un-introspectable by standard tooling
I have yet to see a sysadmin write a 100% correct systemd service unit file
But look, having a stack that replaces many Unix services, and which you basically completely control, is very convenient. It allows, say, to drastically limit third-party userland in embedded systems, where Red Hat is big.
Systemd is trying to replace Unix, a bit more successfully than GNU Hurd.
the whole "sysv init is so slow because of shell scrips" was a scapegoat. yes bash is relatively slow. and yes, dash is the answer to that problem, not systemd.
they who control systemd now control Linux as an OS. not as an API (that's kernel/libc) but as an OS. how you manage it, run it, suspend it, initialize it, turn it off, everything
Only if we ignore the fact that commercial UNIXes already did this before systemd came to be.
Also in the Cloud OS world, classical UNIX doesn't even matter that much, we only need something to run those containers or managed runtimes on top of.
Wayland has a massive loss in functionality (in the name of "security" and off-loading implementation details to compositors/window managers) compared to X11 though.
Stuff like xdotool, screen sharing, clipboard sharing, etc. is much harder.
It works fine on Arch with the Hyprland compositor: putting the mouse from the terminal to waybar, you can still type in the terminal.
When was the last time you really tried wayland? My first (and last) time was 2017. Much has changed!
In a few years, when wine support is perfected, I think people dissing wayland will be seen as quaint as those insisting on a distribution "unsoiled" by systemd are seen today :)
I tried it up to last month in Ubuntu 22.04. I changed back because I got tired of the broken focus-follows-mouse and Ubuntu tracks the bug but doesn't fix it.
Neither. Screensharing was a massive issue for a while, but it's now far easier than X11 because the server/portal is responsible for a lot of the more fiddly bits.
The underlying problem is that the protocol (ABI, really) has to be specified and implemented for these things to work. Screensharing was a prominent early example of something which hadn't been through that process. It was a very visible issue, an easy wound for hardcore X11 fanatics to pull at, and - apparently - continue to bash on long after it has been solved.
Clipboard woes have likewise been solved.
Mouse and keyboard injection has not yet been solved, but I recall there being some draft spec to that effect.
Ultimately, if Wayland does 100% of what you need, you should be using it because it generally does those things better. If it doesn't do what you need, then stick with X11 until it does (and support for that protocol is widespread).
Why not both? It's a loss in functionality because it's much harder and the Linux desktop environment ecosystem is too niche, fragmented and underfunded to make the transition complete in a way that satisfies all camps so we end up in this traditional linux situation where several solutions coexist in parallel forever.
Time heals all wounds. If it is hard but not impossible eventually the gaps will be filled. Also the people who maintain X11 can't live forever sadly. Eventually there will only be Wayland.
We only have to wait a few more decades at the outmost.
The problem is that systemd vs sysv-init is a false dichotomy.
Systemd took over a ton of important non-init functionality, like DNS, logging, and interactive sessions. That could be fine if systemd did so in a nice and rock-solid way, but it was unpleasantly bug-ridden for years after being thrust on mainstream distros via a hard Gnome dependency.
SysV-init sucks in many ways, it's well known, and I can personally attest to that. I don't want SysV-init to be perpetuated.
There were, and are, viable alternatives to both, which are a less radical rework of the well-established Unix approaches around the area, do not overreach well past the init system scope, and are sane and well-functioning. For examples, see [upstart], [openrc], [s6], [runit].
I think that the prevalence of systemd was mostly achieved not through its technical merits (which undeniably exist) but through Red Hat's strongarming, because Red Hat wants certain things work in a way convenient for their business, and they have a powerful battering ram under their control, the Gnome DE.
Fortunately Wayland is not being force-fed in such a way, because, much like systemd when it was introduced, it's still in many important regards not exactly ready. I, with my 25 years of running Linux on desktop, will gladly migrate to a better graphics architecture when in becomes adequate for my purposes, if the whole current Unix architecture is not obsoleted and replaced wholesale by that time.
openrc: I don't believe openrc had a process supervision story 8-9 years ago. from what I recall at the time openrc was just a slightly better sysvinit.
runit: I have a lot of experience with runit, and while it works, it has many footguns and is hard to use correctly.
s6: a better runit with better footguns
A great thing that more people should read, is https://lwn.net/Articles/578210/ from one of the Debian developers who voted to switch to systemd. It touches on things like the lack of adoption of upstart, and openrc not solving the problems they had.
The largest issue with Wayland is it has "Linuxisms". That means no work was done by the Wayland people to make it portable to other UN*X. So the BSD folks (and other UNIX people) have a lot of work to get it going.
And there still seems to be confusion if or will Wayland require systemd, from what I have seen, no 100% clear direction from anyone.
This used to be true, but not anymore. We have upstream support for FreeBSD in libwayland now. Other BSDs have MRs which are only waiting for CI support.
Wayland itself has nothing to do with systemd, many users are running Wayland without systemd.
Yes I can confirm that Wayland runs fine without systemd!
I am on Gentoo and using openrc and sway runs fine. The only thing that might kind of rely on systemd is sway-idle which wants logind (or to run as root iirc).
I know you know this, being the sway and wlroots maintainer and all but I figured I would show a concrete example of this working fine.
FWIW, swayidle only has an optional dependency on logind. It can be built without. In fact, we're even discussing removing the logind-related features from swayidle. (swayidle never required running as root though.)
I mean, is it really wrong for linux people to focus on linux? What's the point of these different OS's if they all support the same software and software has to be made for lowest common denominator?
What a typical Linux user mindset! That IS one of the largest issues for non-Linux users. Linux is by large margin a niche OS in desktop market share. So it is no issue if all programs are developed only for Windows and/or Macintosh. Right?
I admit that appealing to capitalism isn't likely to advance the Unix desktop but the "we're all in this together" excessive portability attitude didn't produce good software either.
For me personally X11 just got to the point where everything I want works out of the box so switching to wayland seems like entirely pointless experience.
I'm also not a fan of attitude of moving support of even basic functionality like "take a screenshot" or "handle mouse/trackpad properly" into higher layers, it just feels like a lot of work duplication and moves that repetition of work to window managers
Yeah, I'm a big fan of systemd but I can accept that when it first came out it was probably a lot worse than it is today. Similarly, Wayland has a bunch of good ideas, along with a bunch of functionality that isn't there yet. Of course, Wayland also has the problem that it's intentionally excluding some useful features that X had, like global hotkeys and screensharing. I think systemd and Wayland are actually opposites in this regard, where systemd was disliked at first and then turned liked, while Wayland was liked at first but is now turning to disliked
> Of course, Wayland also has the problem that it's intentionally excluding some useful features that X had, like global hotkeys and screensharing...
Screen sharing was always a weak point in X11, actually. VNC servers like x11vnc use absurd, inefficient hacks like periodically capturing fragments of the screen and checking for changes [1] -- while an extension to provide change notifications exists, it's been unreliable for ages [2] and the standard advice is to disable it.
> Yeah, I'm a big fan of systemd but I can accept that when it first came out it was probably a lot worse than it is today.
The first "large" distro to switch from sysvinit to systemd was Arch, and that switch happened over ten years ago. The switch itself was quite rocky (the upgrade path was not particularly seamless, and while Arch users tend to be more tolerant of that sort of thing, it's worth mentioning).
That said, even in 2012-2013, the end result once you completed the upgrade was significantly better than collectively expected. The original plan was to support both systemd and sysvinit (at least for a period of time), but that was quickly abandoned because not enough people wanted to actually maintain sysvinit packages, so support[0] ended up getting dropped very quickly.
[0] Arch is a community project, so "support" is different from what you'd expect in (e.g.) RHEL, but it still has separations of what's considered supported and what's not.
> The first "large" distro to switch from sysvinit to systemd was Arch
No. In terms of released to users, the first was Fedora; the second was Arch; then Mageia; then openSUSE. In terms of integrated into the distribution, the first was Fedora; the second was Mageia; then openSUSE; then Arch.
> Of course, Wayland also has the problem that it's intentionally excluding some useful features that X had, like global hotkeys and screensharing
Screensharing has been supported for some time already. Some apps support it, some don't. It is up to the apps to use the respective APIs, the times of free reign over framebuffer is over.
I mean, you can get screen sharing to work. But there are like 3 different incompatible "standards" for how to do it. There's no single simple answer to "how to record the screen in Wayland". This has been the state of screen sharing on Wayland for at least 7 years.
I'm also very curious about what's envisioned for global hotkeys. Surely we don't expect people to manually go to their system settings and configure some command to run which talks to Discord over an IPC solution to start sharing my voice when I press my push to talk button and stop when I release the push to talk button? But "global hotkeys should be configured on a system basis, not an application basis" seems to be the reigning philosophy, despite being incredibly user and developer hostile.
There is one standard and single simple answer: xdg-desktop-portal with pipewire. Some compositors might have implemented their own private APIs, but that is not a standard by definition.
Global shortcuts are a bit more thorny, exactly for the reason you mentioned. You present one POV, the another is, that application-defined shortcuts are incredibly hostile, as they allow application to stomp on each other in the better case, or hijaack global state in the worse one. Some other operating systems do not allow it either, for the same reasons. The long-term solution could be defining api, that allows application to advertise global actions, and allow the user to configure shortcuts that might (or might not) call these, in some user-friendly way.
> A portal frontend service for Flatpak and possibly other desktop containment frameworks.
When it comes to global shortcuts, I'm not saying it has a super easy solution, but it's something that it's essential to support. Wayland intentionally doesn't, and I can't see that changing in the short term (as you also agree)
It is dbus api, and is able to work cross-namespaces (i.e. flatpak containers too). There no harm in using it in non-flatpak apps, at least you will be ready if your app ends up in flatpak.
Wrt global shortcuts, I see that there is some work done. The intentional part isn't malice, as in not willing to implement it at all. It is about not implementing temporary solution, that will be quick and dirty, and then being stuck for supporting it for next 50 years.
It took Golang 12 years to attain generics. Wayland is already 14 years old.
To tell the truth, X11 took roughly 10 years (1986 to 1996) to get to a pretty usable, while pretty imperfect, state, and largely dominate Unix desktops.
I don't see why a there cannot exist a daemon that handles these IPCs in a standard way (e.g. a pipe), and which can be politely asked by apps to map particular hotkeys. An API / UI should be available for the user to review and customize the mapping, and to resolve possible conflicts.
This is basically the approach that exists in macOS for many years; I haven't heard a ton of criticisms towards it.
As you can see with screen capture, even if the api is available, but it will take years to adopt, with some intentionally dragging their feet -- it is different after all, and the old way worked for me, etc, etc.
MacOS ecosystem moves much faster in this regard; mac users expect rapid adoption of new apis, and do not have 20+ years old bash scripts that should continue to work untouched.
> Screensharing has been supported for some time already. Some apps support it, some don't. It is up to the apps to use the respective APIs, the times of free reign over framebuffer is over.
See, that's the problem, it's putting unnecessary load on everything else using it.
It should have that API and it should have API that just allows dedicated app to allow/deny permissions to use that API. Not put everything on WM to duplicated more and more code for no good reason and making WM developer harder
For the past few months I've been investigating and working on porting parts of Xfce to be usable under Wayland. It's astonishing the number of features that are just not implementable at all on Wayland, at least not without inventing new non-standard Wayland protocols. (Another option is refactoring the desktop environment so all the individual components run in the same process as the compositor, and have access to the compositor's internals, but that's unacceptable for what are hopefully obvious reasons.)
Even after over a decade, Wayland still seems quite immature and poorly thought-through. The protocol standardization seems geared toward satisfying GNOME's use-cases and ignoring everyone else's. The wlroots camp has gone their own way on a bunch of things, which is fine, but fractures the landscape a bit.
Making Xfce fully Wayland means turning the window manager, xfwm4, into a Wayland compositor. For someone already familiar with xfwm4's code base, that's a year or more of work (and a requirement we'd have is that it would have to support both X11 and Wayland, further complicating things).
Even if fixing inherent problems in X11 (security, graphics rendering, etc.) would require compatibility breaks, personally I would find that preferable to throwing the entire thing out and having to build (and build on) an entirely new system. As much as I disagree with JWZ's attitude on a lot of things, his description of most open source projects as a "Cascade of Attention-Deficit Teenagers" seems to be pretty accurate, at least here. X11 hasn't been improved and fixed because no one wants to maintain and improve X.org anymore, and the people who used to maintain it would prefer the fun of chasing and working on new shiny things, even if it means decades of new make-work for anyone working in the Linux GUI space.
Don't get me wrong, I am the first one to cut off anyone at the knees who feels they are entitled to tell open source developers what to do with their time (though it gets a bit more complicated when many of those developers are employed by corporations to do this work). But I think it's pretty shitty to push the desktop in this direction and essentially force desktop and toolkit and application developers to choose between stepping up to maintain and build on X.org (something well out of most people's wheelhouse), or spend a huge amount of time porting to a new display system.
Having said that, Wayland does have promise to be a better system than X11, even though it will likely take another decade to achieve feature parity with what we already have. So I'll continue to work on getting there eventually, even though I resent the fact that I have to learn an entirely new display system so I can work on reimplementing the same features again instead of building new features, fixing bugs, and making things more polished.
> Even if fixing inherent problems in X11 (security, graphics rendering, etc.) would require compatibility breaks, personally I would find that preferable to throwing the entire thing out and having to build (and build on) an entirely new system.
Why?
X11 in the current state would need to be radically redesigned anyway.
There's a bunch of stuff that long stopped making sense, like XDrawLine and similar. The networking protocol sucks horribly and just doesn't perform, even on modern, high end connections, and there's a bunch of baked in assumptions that don't match modern hardware.
Yeah, you could make X12 break compatibility, throw out all the cruft, and redesign the protocol, but at that point, what is even the point? It'll break pretty much every single application in existence anyway.
Some people apparently really do hate it when things change that they seemingly have no control over
Edit: I fully expect to be downvoted into oblivion for this post :-D