Cleanup Systemd Journald Storage

With the move from sysvinit to systemd, there were lots of small but important changes to the Linux ecosystem. One of them was the move from traditional syslog daemons to Systemd Journald. Now I’m not going to say this is a good or bad thing, as it entirely depends on your old habits and new optimism. What it does mean is a move to a faster and more flexible system log format but at the cost of some added complexity.

Historically, logrotate could be used to squash log files that got big over time. However, since Systemd Journald stores logs in a new binary format that approach doesn’t really work anymore. It’s best to let it do it’s own thing. Sometimes, though, you need to step in…

First, the wrong way:

The wrong thing to do is delete the files in /var/log/journal. Since the systemd journal is essentially a database, this is a pretty bad idea. It’s probably going to be okay, but it’s so easy to do it correctly that it’s not worth the risk.

Now, the right way:

Purge logs by size

Here’s a new command I learned today; how to force journald to manually squash the logs:

journalctl --vacuum-size=500M

And just like that, journald will purge old logs until it’s under the size you requested. In my case, my Linux laptop was using over a gig of space in /var/log/journal, which is a sizable portion of my frugal 120 gig solid state drive. Here’s what that looked like:

$ sudo journalctl --vacuum-size=500M

Deleted archived journal /var/log/journal/f8dfbb5a095c4855a5f326d99bb598a6/system@00058074d01b8d87-fe43a5c3320c2b97.journal~ (24.0M).
....
Vacuuming done, freed 672.1M of archived journals from /var/log/journal/f8dfbb5a095c4855a5f326d99bb598a6.

Purge logs by date

Alternatively, it might be desired to only retain a few days of logs. I don’t really mind losing some data retention from my beater laptop, so this isn’t an issue. Obviously, you shouldn’t do this on a server unless you have a reliable log collector. I only need the last two days of logs, so let’s see how much space can be reclaimed:

$ sudo journalctl --vacuum-time=2d

Deleted archived journal /var/log/journal/f8dfbb5a095c4855a5f326d99bb598a6/user-1000@00058a8950e821fe-033394b0a263524f.journal~ (16.0M).
....
Vacuuming done, freed 488.1M of archived journals from /var/log/journal/f8dfbb5a095c4855a5f326d99bb598a6.

After chopping off all those logs, the entire journal is only taking up 33 MB!

$ sudo du -cksh /var/log/journal/*

33M	journal/f8dfbb5a095c4855a5f326d99bb598a6
8.0K	journal/remote
33M	total

Hopefully this can help somebody else adjust to the new way of logging!