Running systemd without systemd-journald

TLDR
All I needed was:

patch /etc/systemd/journald.conf << EOF
@@ -12,7 +12,7 @@
    # See journald.conf(5) for details.

    [Journal]
-#Storage=auto
+Storage=none
    #Compress=yes
    #Seal=yes
    #SplitMode=uid
EOF
systemctl restart systemd-journald
rm -rf /var/log/journal/*

But I tried to pull out this weed. I'd better spent wasted time on looking for a distro without systemd. And without sysvinit. I dislike that more than systemd.

Finally, I got really angry. On all my systems the top memory consumer is systemd-journald. On my Orange/NanoPI home servers with 2GB RAM it's not a big deal. However, it depends. Given the number of LXC containers, it really is. Things get much worse on a VPS with 512Mb RAM, where 98Mb is consumed by that shit.

Even if I restart systemd-journald, in few seconds it becomes among top 3 memory consumers.

I asked myself: when I needed systemd binary logs? - and I realized I never needed them. I'm happy with old good text-based logs and I always install rsyslog if it is not present in the base system.

I quickly searched for an existing solution how to delete systemd-journald but the freshest relevant one was dated 2015 and people were unsuccessful in their attempts.

Someone said systemctl wouldn't print nicely looking statuses, but I even don't remember when I needed it to check status of services. Manually, I mean. Instead, I use ps aux, I look into /var/log/syslog, but I hardly can remember I ever ran systemctl status to admire its output.

The best solution would be changing linux distro, but that's not easy. I'm quite conservative. Debian and Armbian is my choice for now and I'm not ready to change it. Although Debian is getting lousy, I dislike other distros much more. Nevertheless, I keep looking for replacement. But right now, instead of devoting my time to painting, I have to urgently dive in shit.

Basically, I'm not against systemd. I find its units quite convenient and the babysitter approach imho is much better than the mess with PID files. But if I need to fix something, it's much more difficult with systemd than with sysvinit.

By the way, why did they buried upstart? I was quite happy with it in old good times when Ubuntu was sane.

Back to journald, I tried to play in a LXC container first. Actually I played with a couple of my containers. Started from absolutely unnecessary one, where I simply deleted /lib/systemd/systemd-journald executable (this did not render the container inoperational, though), I finished in another, slightly more important container. All the notes below are based on more sane approach, without touching base system files.

Here's what I did in the first place:

systemctl stop systemd-journald

It printed:

Warning: Stopping systemd-journald.service, but it can still be activated by:
systemd-journald-dev-log.socket
systemd-journald.socket
systemd-journald-audit.socket

Good to know. I'll use this knowledge later on.

But now, is the system still operational? Yes, it is.
Does anything go to /var/log/syslog? Yes, it goes.
So far so good? - No. It gets resurrected very soon.

What's next? Let's try to disable it completely:

systemctl mask systemd-journald
Created symlink /etc/systemd/system/systemd-journald.service → /dev/null.

shutdown -r now

My container got restarted without problems, but it took more time than usual.

Checking output from ps aux, I see no systemd-journald However, nothing goes to /var/log/syslog That's because of missing /dev/log which is needed by rsyslog.

To fix this, let's review units printed by systemctl stop systemd-journald. The only of them, namely /lib/systemd/system/systemd-journald-dev-log.socket, has interesting stuff.

Normally, if we can say so about systemd, the listening socket is located at /run/systemd/journal/dev-log and /dev/log is a symlink to it.

We don't need that, so:

systemctl mask systemd-journald-dev-log.socket
Created symlink /etc/systemd/system/systemd-journald-dev-log.socket → /dev/null.

Rsyslog depends on syslog.socket unit that creates /run/systemd/journal/syslog socket. The reliable method to fix that is to copy /lib/systemd/system/syslog.socket to /etc/systemd/system/ and change socket path (ListenDatagram option) to /dev/log.

Rebooted.

It works. Not like a charm, though. Remaining systemd components complain about missing journal socket. I don't think it's worth to play with my production VPS same way.

I don't want to try this on a KVM container.
I don't want to try this on another my laptop.

I give up. Enough swimming in shit, I'm awaited by more interesting things.

But for LXC this seems to work. However I wouldn't use this approach in production.