Linux: Fix bug 12309

In this article I will tell you about the most known bug in Linux distributions.

Most likely, during your time using Linux, you have at least once faced the problem that when you run a large number of programs or heavy loads on the file system, the computer “freezes”, often to the point of no return, when nothing but a forced reboot helps. This is a consequence of Linux kernel bug 12309.

Linux

The source of the problem is the high load I/O bus. In addition to user actions, data on the disk is regularly unloaded/loaded from RAM to a SWAP partition.

OOM (Out Of Memory) is a sutuation when all RAM space is full and new processes can’t be loaded.

Swapping can be completely disabled, but when the system will run out of memory, the built-in OOM-killer will simply kill most aggressive processes, and you will get force closed applications with unsaved data.

Despite the fact that this problem has been known since 2008, it hasn’t been completely fixed by the developers. Users are offered several ways to fix it.

ZSWAP and ZRAM

The ZRAM kernel module creates the swap partition directly in RAM. This solution is used, for example, in Android. Unnecessary memory pages are run through the compression algorithm and take up less space, and are decompressed back if necessary. ZRAM can increase the load on the CPU, but when using an efficient compression algorithm and a properly selected ratio, this increase in load isn’t critical.

ZSWAP is an intermediate buffer before writing to the swap partition. As soon as the ZRAM swap is full, ZSWAP will connect.

We need to install the systemd-swap utility. Enter the command:

sudo apt-get install systemd-swap   # Debian-based
sudo dnf install systemd-swap       # RHEL-based
sudo pacman -S systemd-swap         # Arch-based

Then we need to configure GRUB. Enter the command:

sudo nano /etc/default/grub

We need to edit GRUB_CMDLINE_LINUX. Add these arguments:

zswap.enabled=1 zswap.max_pool_percent=15

The first argument is responsible for enabling the ZSWAP module, and the second for the percentage of RAM allocated for the ZSWAP space. The recommended value is 15.

Press ctrl+O to confirm the changes, confirm the file path by pressing Enter and exit by pressing ctrl+X.

Now we need to rebuild GRUB. Enter the command:

sudo grub-mkconfig -o /boot/grub/grub.cfg

To configure ZRAM, we will use udev rule. For this operation we need superuser rights:

sudo su

First, add the kernel module to startup. Enter the command:

echo zram > /etc/modules-load.d/zram.conf

Second, we need to configure ZRAM rules. Enter the command:

echo KERNEL=="zram0", ATTR{disksize}="512M" RUN="/usr/bin/mkswap /dev/zram0", TAG+="systemd" > /etc/udev/rules.d/99-zram.rules

Attention! Use /usr/sbin/mkswap path for Debian.

The command above allocates a new swap partition with 512Mb size.

Then we need to add ZRAM-patition to fstab. Enter the command:

sudo echo /dev/zram0 none swap defaults 0 0 >> /etc/fstab

Reboot your PC.

EARLYOOM

Earlier in the article, it was mentioned that the kernel has a built-in tool for resolving OOM situations. However, it doesn’t always help, since it only works after a problem occurs, which may be too late.

Earlyoom is a small utility which monitors the memory and swap load, and starts killing the processes before 12309 occurs. First you need to install the utility:

sudo apt-get install earlyoom   # Debian-based
sudo dnf install earlyoom       # RHEL-based
sudo pacman -S earlyoom         # Arch-based

It works as a systemd service. Configure earlyoom:

sudo nano /etc/default/earlyoom

Let’s look the config file:

Earlyoom config file

In this file, we are interested in the EARLYOOM_ARGS line. Let’s look the arguments:

1. -m specifies the minimum amount of RAM that should remain free, in percentage.
2. -s specifies the amount of Swap space
3. --prefer is a regular expression. The processes that satisfy him will be closed first.
4. --avoid has the opposite meaning. It usually indicates X server processes, systemd and other background system services and important processes.
5. -r specifies the frequency of checking, in seconds

Editing is strongly for yourself. In most situations the default config is good.

Press ctrl + O to confirm the changes, confirm the file path by pressing Enter and exit by pressing ctrl + X.

Then we need to restart the earlyoom service:

sudo systemctl restart earlyoom

I/O Scheduler

Another way is to change the disk I/O scheduler. The default scheduler isn’t so efficient one and is a good idea to replace it. To check which scheduler is currently standing, you need to enter this command:

cat /sys/block/<device>/queue/scheduler

Where <device> is a disk’s name (for example, sda, sdb, nvme0n1, hda, hdb, etc)

The selected scheduler for /dev/sda is mq-deadline in square brackets

We need superuser rights to configure udev rule. Enter the command:

sudo su

Create the config file. Enter the command:

nano /etc/udev/rules.d/60-ioschedulers.rules

Then we need to configure schedulers for every type of disks. Insert this code to the file:

# NVMe
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# SATA SSD and eMMC
ACTION=="add|change", KERNEL=="sd[a-z]|mmcblk[0-9]*", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# Rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"

In this file, there are 3 types of disks:

1. NVMe — no scheduler needed
2. SATA SSD and eMMC — mq-deadline
3. Rotating disks (HDD, DVD, Floppy, etc.) — BFQ

Press ctrl + O to confirm the changes, confirm the file path, take Enter and exit by pressing ctrl + X.

Reboot your computer.

Sources

This article is based on my previously made article in VK Unix Power group.
Source