Raymii.org
Quis custodiet ipsos custodes?Home | About | All pages | Cluster Status | RSS Feed
Limit specific process memory on desktop linux with cgroups and earlyoom
Published: 13-02-2021 | Author: Remy van Elst | Text only version of this article
❗ This post is over three years old. It may no longer be up to date. Opinions may have changed.
Table of Contents
On my laptop I recently had trouble with out of memory issues when running clion, firefox, thunderbird, teams and a virtualbox VM. To combat this, I've setup cgroups to limit how much RAM specific applications can use and configured earlyoom, a very nifty tool that checks available memory and kills the process with the highest oom_score if available memory falls below 5%. Otherwise, my laptop would first grind to a halt (even without swap) and only after half an hour of seemingly being stuck would the OOM killer kick in. With earlyoom this hanging behavior is gone, although sometimes applications get killed when I don't expect it. I've given firefox, thunderbird and teams a cgroup with memory limit and clion and virtualbox use their own configuration to limit their RAM usage.This post details how to setup cgroups to limit memory of specific processes including automatically placing process inside a cgroup.
I'm using Microsoft Teams in this example, that abomination of a chrome-browser / glorified IRC client has hardware requirements stating at least 4 GB of RAM, and that is way too much for what it's worth. Even my java-based CLion IDE doesn't use as much memory as Microsoft Teams. I've now given it 2 GB RAM max and it works just fine. 1.5 GB RAM also works in my experience. I think Teams has a memory leak because it starts with around 600 MB in use, but after running for a few hours, it is up and over 1 GB, eventually being killed by the OOM-killer in the cgroup. After restarting it's back around 600 MB again.
Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:
I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!
Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.
You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!
This post is tested on Ubuntu 20.04. As far as documentation tells me, it
should work on Debian 10 and Ubuntu 18.04 as well, but I've not tested that.
Documentation on cgroups is spread out and not coherent. When looking up
documentation, make sure you know which cgroup version you use and for what
cgroup version the guide is written. I try to avoid systemd-specific cgroup
configuration (slices) since this post should also be applicable for
non-systemd users. Therefore you'll see me poking around in /sys/fs/cgroup/
instead of using systemd tools.
I've ended up with the following RAM limits:
- Teams: 2 GB
- Firefox: 2 GB
- Thunderbird: 1 GB
- CLion: 4 GB
- Virtualbox VM: 2 GB
About 11 GB reserved, leaving a bit for all other applications such as the
desktop. This limited configuration runs for a few days and I haven't had the
out of memory issues. Using munin and the multips plugin I can see that
the processes stay inside their given limits. Here is the munin graph that
shows the specific processes I monitor:
java is CLion, and since the machine is not on all the time, there are gaps
in the graph, but it is more than enough to give a general overview of usage.
The default memory usage graph is fun to look at as well:
Earlyoom, a more desktop friendly oom-killer
Starting off with earlyoom, as of Debian 10 and Ubuntu 18.04, available in the repositories, install it using the package manager:
apt install earlyoom
You don't have to do anything more, when your memory usage drops below 10% it
will start killing processes, by default the one with the highest oom_score.
In my case that was often teams, but sometimes it was firefox. I've changed
the settings to only kick in when memory usage drops below 5% and have added a
few processes which I'd rather not have killed. kwin is the KDE window
manager, when that was killed my window borders were gone. Fixed by a kwin
--replace, but annoying. VirtualboxVM killing could give disk corruption
inside the VM, which is something I'd also rather avoid.
vim /etc/default/earlyoom
Contents:
# Print every 60 seconds, act if free memory comes below 5% and avoid killing KDE and virtualbox
EARLYOOM_ARGS="-r 60 -m 5 --avoid '(^|/)(kwin_x11|kwin|ssh|VirtualBoxVM)$'"
Limiting memory per process with cgroups
cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. It's the magic behind linux containers (lxc/docker) and I'm using it to make sure specific processes and their children cannot allocate more than a given amount of RAM. On Ubuntu you must install the following package:
apt install cgroup-tools
Create a cgroup, I named mine cgTeams:
sudo cgcreate -t remy:remy -a remy:remy -g memory:/cgTeams
Replace remy:remy by your username and group.
Set the maximum amount of RAM for the newly created cgroup. The calculation to
bytes isn't required on modern linux, you could just enter 2048m but on older
Debian systems you do need to specify the exact bytes:
echo $(( 2048 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.limit_in_bytes #2 GB RAM
If you have swap enabled, you can set a limit on that as well:
echo $(( 2049 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.memsw.limit_in_bytes #2GB swap, only works if you have swap
Launch Teams in the freshly created cgroup:
cgexec -g memory:cgTeams teams
In the next few sections I'll discuss automatic creation of cgroups at boot and automatic placement of processes inside cgroups. If all you wanted was to give a specific process a RAM limit, not making it persistent, this is all there is to it.
What happens when a process tries to allocate more RAM than it is allowed?
On modern systems, the OOM killer will kill the cgroup-ed process. Quoting the kernel documentation on this one:
When a cgroup goes over its limit, we first try to reclaim memory from the cgroup so as to make space for the new pages that the cgroup has touched. If the reclaim is unsuccessful, an OOM routine is invoked to select and kill the bulkiest task in the cgroup.
I first tried to give Teams one gigabyte of RAM, which wasn't enough. Teams
showed the splash screen and failed to start, dmesg -T showed me that it was
killed right away inside the cgroup:
[Thu Feb 11 12:46:42 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgTeams,task_memcg=/cgTeams,task=teams,pid=22920,uid=1000
[Thu Feb 11 12:46:42 2021] Memory cgroup out of memory: Killed process 22920 (teams) total-vm:2513452kB, anon-rss:322756kB, file-rss:57980kB, shmem-rss:0kB, UID:1000 pgtables:3444kB oom_score_adj:300
[Thu Feb 11 12:46:42 2021] oom_reaper: reaped process 22920 (teams), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Firefox behaves fun as well with a RAM limit. Large webpages get killed and a funny message is shown:
dmesg -T shows that it is not the firefox process, but a child named Web Content:
[Fri Feb 12 12:49:50 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgFirefox,task_memcg=/cgFirefox,task=Web Content,pid=98779,uid=1000
[Fri Feb 12 12:49:50 2021] Memory cgroup out of memory: Killed process 98779 (Web Content) total-vm:4292056kB, anon-rss:1657332kB, file-rss:108680kB, shmem-rss:95364kB, UID:1000 pgtables:8884kB oom_score_adj:0
[Fri Feb 12 12:49:50 2021] oom_reaper: reaped process 98779 (Web Content), now anon-rss:0kB, file-rss:0kB, shmem-rss:94228kB
Oh well, at least my machine doesn't crash when I open a large merge request on gitlab with hundreds of changes.
Activate cgroups at boot with cgconfigparser
When this is all working you can make it permanent by placing this
configuration in the file /etc/cgconfig.conf:
group cgTeams {
perm {
admin {
uid = remy;
}
task {
uid = remy;
}
}
memory {
memory.limit_in_bytes = 1585446912;
}
}
Test the file for syntax errors:
sudo cgconfigparser -l /etc/cgconfig.conf
The command should not give any output. For reference, I have three groups configured:
group cgTeams {
perm {
admin {
uid = remy;
}
task {
uid = remy;
}
}
memory {
memory.limit_in_bytes = 2048m;
}
}
group cgFirefox {
perm {
admin {
uid = remy;
}
task {
uid = remy;
}
}
memory {
memory.limit_in_bytes = 2048m;
}
}
group cgThunderbird {
perm {
admin {
uid = remy;
}
task {
uid = remy;
}
}
memory {
memory.limit_in_bytes = 2048m;
}
}
systemd service for cgconfigparser
On Ubuntu 20.04 there is no systemd service to start cgconfigparser at
boot. Here is a relatively simple service file I use to start
cgconfigparser:
vim /lib/systemd/system/cgconfigparser.service
Contents:
[Unit]
Description=cgroup config parser
After=network.target
[Service]
User=root
Group=root
ExecStart=/usr/sbin/cgconfigparser -l /etc/cgconfig.conf
Type=oneshot
[Install]
WantedBy=multi-user.target
Enable and start the service:
systemctl enable cgconfigparser
systemctl start cgconfigparser
On boot the cgroups will then be created via this service. Easy to adapt to
other init systems like openrc.
You can continue on with the next section to automatically place specific
processes into specific cgroups via cgrulesengine but, for a simpler
solution, you can edit your desktop launcher to run the command prefixed with
cgexec. Or a simple cronjob every minute that runs cgclassify -g
memory:cgTeams $(pidof teams) to put every running teams process in that
cgroup. Before I had set up cgrulesengined I used three cronjobs every
minute running cgclassify.
Automatically put processes into a specific cgroup with cgrulesengined
This part was a bit vague online, but there exists a daemon that automatically puts processes inside cgroups based on a few rules. The startup scripts in most versions of Debian and Ubuntu are either broken or missing, but the daemon seems to work. The manpage had more information about the syntax, I'll limit the example to our usecase. Edit the following file:
vim /etc/cgrules.conf
Contents:
remy:teams memory cgTeams/
remy:firefox memory cgFirefox/
remy:thunderbird memory cgThunderbird/
Replace remy by your username, the processes (teams) and the cgroups
(cgTeams) by your own. Save and check the file with:
/usr/sbin/cgrulesengd -vvv
No output means no errors. You can omit :processname to limit everything by
a user, memory can be replaced by other cgroup categories like cpu, but in
our situation that is not applicable. I want to limit the memory of a few
specific applications, not CPU cores or other resources.
systemd service for cgrulesengined
Just as with cfconfigparser, there is no default service on Ubuntu, but adding one is just as simple as the previous one.
Copy one of the configuration files:
cp /usr/share/doc/cgroup-tools/examples/cgred.conf /etc/cgred.conf
Add a systemd unit file:
vim /lib/systemd/system/cgrulesgend.service
Contents:
[Unit]
Description=cgroup rules generator
After=network.target cgconfigparser.service
[Service]
User=root
Group=root
Type=forking
EnvironmentFile=-/etc/cgred.conf
ExecStart=/usr/sbin/cgrulesengd
Restart=on-failure
[Install]
WantedBy=multi-user.target
Enable and start the service:
systemctl enable cgrulesgend
systemctl start cgrulesgend
After a reboot you should have the processes you configured in
To check if the process actually launches in the correct cgroup after a reboot
you need to use the cgroup filesystem. The file
/sys/fs/cgroup/memory/cgTeams/tasks lists all the process ID's that run in
that cgroup. I can see that it is working for teams:
$ for pid in $(cat /sys/fs/cgroup/memory/cgTeams/tasks); do pgrep $pid; done
remy 3305 2.4 2.0 3688372 327456 ? Sl 06:47 3:18 /usr/share/teams/teams
remy 3307 0.0 0.2 189960 39656 ? S 06:47 0:00 /usr/share/teams/teams --type=zygote --no-sandbox
remy 3367 0.0 0.5 1832608 87364 ? Sl 06:47 0:00 /usr/share/teams/teams --type=renderer