Linux OOM Kill With Free RAM Left? Read This

free says 8 GB available, and yet your database just got shot by the OOM killer. Is Linux lying? No, but it is answering a different question than the one you asked. This guide explains why a Linux box can hit an out-of-memory kill while RAM still looks free, and gives you the commands to read the real numbers: MemAvailable, slab, overcommit accounting and oom_score_adj.

Four different meanings of out of memory

When a process dies with 'out of memory', one of four things usually happened. Knowing which one saves hours of guessing.

• Real exhaustion: anonymous memory (heap, stacks) plus unreclaimable kernel memory ate everything, and swap, if any, is full.
• Unreclaimable kernel memory: slab objects that cannot be freed, so the page cache cannot save you.
• A limit smaller than the machine: a cgroup (container, systemd service) hit its own memory.max, so the system looked fine while the group did not.
• Strict overcommit: allocations failed because the commit limit was reached, even though pages were physically free.

Step 1: read /proc/meminfo like a detective

The kernel documentation defines MemAvailable as an estimate of how much memory is available for starting new applications without swapping, calculated from MemFree, SReclaimable, the file LRU lists and the low watermarks. It deliberately accounts for the fact that the system needs some page cache to function and that not all reclaimable slab will actually be reclaimable, because some items are in use.

grep -E '^(MemTotal|MemFree|MemAvailable|Cached|Slab|SReclaimable|SUnreclaim|KernelStack|PageTables|CommitLimit|Committed_AS|SwapTotal|SwapFree):' /proc/meminfo

# Convert to GiB for humans
awk '/^(MemAvailable|Slab|SUnreclaim|Committed_AS|CommitLimit):/ {printf "%-14s %6.2f GiB\n", $1, $2/1048576}' /proc/meminfo

Per the same document, Slab is the in-kernel data structure cache, SReclaimable is the part that might be reclaimed (such as caches), and SUnreclaim is the part that cannot be reclaimed under memory pressure. If SUnreclaim is huge and growing, you have a kernel-side leak or a driver hoarding memory, and no amount of dropping caches will help.

Step 2: find who owns the slab

slabtop shows which kernel object caches are fat. Typical culprits are dentry and inode caches after a filesystem walk (reclaimable) or networking and driver objects (often not).

sudo slabtop -o -s c | head -20

# Same data without slabtop (needs root)
sudo awk 'NR>2 {printf "%10.1f MiB  %s\n", $3*$4/1048576, $1}' /proc/slabinfo | sort -rn | head -15

The vm.vfs_cache_pressure sysctl controls the tendency to reclaim dentry and inode caches. The kernel docs warn that setting it to 0 means dentries and inodes are never reclaimed under memory pressure, which "can easily lead to out-of-memory conditions". If someone 'optimised' it to 0 years ago, you may have found your culprit.

Step 3: overcommit, the promise that can be broken

Linux lets processes allocate more virtual memory than exists, betting that most of it will never be touched. The kernel docs describe three modes set by vm.overcommit_memory: 0 is heuristic handling that refuses obvious overcommits and is the default; 1 always overcommits; 2 never overcommits, capping total commit at swap plus a configurable share of RAM.

In mode 2 the limit is visible in /proc/meminfo. The documented formula is CommitLimit = (total RAM pages minus huge TLB pages) times overcommit_ratio / 100, plus swap pages. Committed_AS is what has been promised so far. The docs give an example: 1G of RAM and 7G of swap at a ratio of 30 yields a 7.3G limit. Note the trap: the limit is only enforced in mode 2, so in modes 0 and 1 these numbers are informational.

sysctl vm.overcommit_memory vm.overcommit_ratio
awk '/^CommitLimit:/ {l=$2} /^Committed_AS:/ {c=$2} END {printf "Committed %.1f%% of CommitLimit\n", 100*c/l}' /proc/meminfo

# Try strict mode temporarily, then revert (test on a lab box first)
sudo sysctl -w vm.overcommit_memory=2 vm.overcommit_ratio=80

Step 4: how the OOM killer picks a victim

The kernel assigns each candidate a badness score from 0 (never kill) to 1000 (always kill), roughly the proportion of allowed memory the task uses, based on its memory and swap use. If a task uses half its allowed memory, its score is about 500. The 'allowed' memory depends on context: the whole machine, a cpuset, a mempolicy node set, or a memory cgroup limit.

You can read the score and nudge it. oom_score_adj accepts -1000 to +1000 and is added to the badness score. The documentation states that -1000 is equivalent to disabling OOM killing for that task. The value exported in oom_score includes the adjustment, so it ranges effectively over 0 to 2000.

# Top 10 victims right now
for p in /proc/[0-9]*; do
  pid=$(basename "$p")
  [ -r "$p/oom_score" ] && printf "%6s %5s  %s\n" "$(cat $p/oom_score)" "$pid" "$(tr '\0' ' ' < $p/cmdline | cut -c1-60)"
done 2>/dev/null | sort -rn | head -10

# Make the database less attractive, and a batch job more attractive
echo -500 | sudo tee /proc/$(pidof postgres | cut -d' ' -f1)/oom_score_adj
echo 500  | sudo tee /proc/$(pidof my-batch-job)/oom_score_adj

For services, set it declaratively. systemd supports OOMScoreAdjust= in the unit file, and it survives restarts, unlike echoing into /proc.

[Service]
OOMScoreAdjust=-500
# Optional: hard ceiling for this service so it cannot starve the host
MemoryMax=12G

Step 5: read the crime scene

Every kill leaves a report in the kernel log, including a memory summary and a table of tasks with their oom_score_adj. Start there before changing any sysctl.

sudo journalctl -k --since "-2 hours" | grep -iE 'out of memory|oom-kill|Killed process'
sudo dmesg -T | grep -A40 -i 'invoked oom-killer' | head -80

Takeaway: a checklist

• Read MemAvailable, not MemFree.
• Compare Slab, SReclaimable and SUnreclaim; use slabtop on the biggest.
• Check vm.overcommit_memory and, in mode 2, Committed_AS against CommitLimit.
• Check for cgroup limits and memory.events before blaming the host.
• Protect critical services with OOMScoreAdjust= and give them sane MemoryMax values.
• Always read the kernel's OOM report first; it names the constraint.

The 'free says plenty' paradox is rarely a kernel bug. It is a mismatch between a global free-memory number and the specific limit, kind of memory or cgroup that actually ran dry. Measure the right one and the mystery usually solves itself.