A Guide to systemd journal Maintenance [With Examples]

5 min


Systemd comes with many built-in features to manage the system logs. In this guide, we explain how you can manage system journals, logs and take action on them such as rotating, archiving, and clear logs. We also explain the manual systems journal clean method and using config file changes.

If your Linux distribution supports systemd, then it collects logs from all processes, applications of the system every second which starts from the boot. All these logging events are managed by journald daemon of systemd. The journald collects all the logs (info, warnings, errors, etc) and stores them as binary data in the disk files. 

As the logs remain in the disk and every second it is collected, it takes up huge disk space; especially for older systems, servers. For example, in one of my test systems which are running for around one year, the log file size is in GBs.

If you manage multiple systems, servers, it is always recommended to properly manage journald logs for efficient operation. Let’s take a look at how you can manage the log files.

The systemd journal Maintenance

Using the journalctl utility of systemd, you can query these logs, perform various operations on them. For example, viewing the log files from different boots, check for last warnings, errors from a specific process or applications. If you are unaware of these, I would suggest you quickly go through this tutorial – “use journalctl to View and Analyze Systemd Logs [With Examples]” before you follow this guide. 

Where are the physical journal log files?

The systemd’s journald daemon collects logs from every boot. That means, it classifies the log files as per the boot. 

The logs are stored as binary in the path /var/log/journal with a folder as machine id.

For example:

Screenshot of physical journal file -1
Screenshot of physical journal file -1
Screenshot of physical journal files -2
Screenshot of physical journal files -2

Also, remember that based on system configuration, runtime journal files are stored at /run/log/journal/. And these are removed in each boot. 

Can I manually delete the log files?

You can, but don’t do it. Instead, follow the below instructions to clear the log files to free up disk space using journalctl utilities.

How much disk space is used by systemd log files?

Open up a terminal and run the below command.

journalctl --disk-usage

This should provide you with how much is actually used by the log files in your system.

journalctl disk usage command
journalctl disk usage command

If you have a graphical desktop environment, you can open the file manager and browse the path /var/log/journal and check the properties.

systemd journal clean process

The effective way of clearing the log files should be done by journald.conf a configuration files. Ideally, you should not manually delete the log files even if the journalctl provides the utility to do that.

Let’s take a look at how you can delete it manually, then I will explain the configuration changes in journald.conf so that you do not need to manually delete the files from time to time; Instead, the systemd takes care of it automatically based on your configuration. 

Manual delete

First, you have to flush and rotate the log files. Rotating is a way of marking the current active log files as an archive and creating a fresh logfile from this moment. The flush switch asks the journal daemon to flush any log data stored in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled.

Then, after flushing and rotating, you need to run journalctl with vacuum-size, vacuum-time, and vacuum-files switches to force systemd to clear the logs. 

Example 1:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-time=1s

The above set of commands removes all archived journal log files until the last second. This effectively clears everything. So, be careful while running the command. 

journal clean up - example
journal clean up – example

After clean up:

After clean up - journal space usage
After clean up – journal space usage

You can also provide the following suffixes as per your need following the number.

  • s: seconds
  • m: minutes
  • h: hours
  • days
  • months
  • weeks
  • years

Example 2:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-size=400M

This clears all archived journal log files and retains the last 400MB files. Remember this switch applies to only archived log files only, not on active journal files. You can also use suffixes as below.

  • K: KB
  • M: MB
  • G: GB

Example 3:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-files=2

The vacuum-files switch clears all the journal files below the number specified. So, in the above example, only the last 2 journal files are kept and everything else is removed. Again, this only works on the archived files.

You can combine the switches if you want, but I would recommend not to. However, make sure to run with --rotate switch first.

Automatic delete using config files

While the above methods are good and easy to use, it is recommended that you control the journal log file cleanup process using the journald configuration files which present at /etc/systemd/journald.conf

The systemd provides many parameters for you to effectively manage the log files. By combining these parameters you can effectively limit the disk space used by the journal files. Let’s take a look.

journald.conf parameterDescriptionExample
SystemMaxUseSpecifies the maximum disk space that can be used by the journal in persistent storageSystemMaxUse=500M
SystemKeepFreeSpecifies the amount of space that the journal should leave free when adding journal entries to persistent storage.SystemKeepFree=100M
SystemMaxFileSizeControls how large individual journal files can grow to in persistent storage before being rotated.SystemMaxFileSize=100M
RuntimeMaxUseSpecifies the maximum disk space that can be used in volatile storage (within the /run filesystem).RuntimeMaxUse=100M
RuntimeKeepFreeSpecifies the amount of space to be set aside for other uses when writing data to volatile storage (within the /run filesystem).RuntimeMaxUse=100M
RuntimeMaxFileSizeSpecifies the amount of space that an individual journal file can take up in volatile storage (within the /run filesystem) before being rotated.RuntimeMaxFileSize=200M

If you add these values in a running system in /etc/systemd/journald.conf file, then you have to restart the journald after updating the file. To restart use the following command. 

sudo systemctl restart systemd-journald

Verification of log files

It is wiser to check the integrity of the log files after you clean up the files. To do that run the below command. The command shows the PASS, FAIL against the journal file.

journalctl --verify
verify log files
verify log files

Closing Notes

I hope this guide helps you to understand the basics of the systemd journal management process. With these, you can manage the disk space used by the log files in your system or servers by limiting the space, clearing old log files. These are just the guideline commands, you can combine these in multiple ways to achieve your system demands. 


We bring the latest tech, software news and stuff that matters. Stay in touch via Telegram, Twitter, YouTube, and Facebook and never miss an update!

Join our Telegram channel and stay informed on the move.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.


Arindam

Creator of debugpoint.com. All time Linux user and open-source supporter. Connect with me via Telegram, Twitter, LinkedIn, or send us an email.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
1
0
Would love your thoughts, please comment.x
()
x