◀ Back to blog
Linux

Linux performance monitoring

Published on 15 Mar 2024· 9 min read
#Linux#Monitoring#Performance

Monitoring system performance

Good monitoring is essential to detect problems before they affect users. A disk filling up, a memory leak in a PHP worker or a process hogging the CPU almost always shows up in the metrics hours, even days, before the outage. You still need to know what to look at, with which tools, and at what threshold to react.

This article covers two complementary levels: live diagnosis, when you are logged in over SSH on a machine that is struggling, and continuous monitoring, with historical metrics and automatic alerts.

A method before the tools

When facing a slow server, the temptation is to start htop and stare at the screen. A more effective approach is to go through each resource (CPU, memory, disk, network) and ask three questions, following the USE method popularised by Brendan Gregg:

  • Utilisation: what percentage of the time is the resource busy?
  • Saturation: is work queuing up for lack of capacity?
  • Errors: is the resource reporting errors (dropped packets, I/O errors)?

A resource can be 100% utilised without any problem, as long as there is no saturation. Conversely, a disk at 60% utilisation with a long queue is already a bottleneck.

Essential tools

# Real-time CPU and memory
htop

# Disk statistics
iostat -x 1

# Network traffic
iftop -i eth0

# Resource-hungry processes
ps aux --sort=-%mem | head -20

A few pointers for reading these outputs:

  • htop shows per-core usage, memory and swap. Sort by column with F6 and display the process tree with F5 to see which parent started which children.
  • iostat -x 1 (sysstat package) refreshes every second. Watch %util (device busy time), r_await and w_await (average latency in milliseconds) and aqu-sz (average queue size). Rising latency signals saturation.
  • iftop shows throughput per connection. Adjust the interface name: on recent distributions it is often called ens3 or enp0s3 rather than eth0 (check with ip -br link).
  • ps aux --sort=-%mem lists the processes using the most memory. Replace it with --sort=-%cpu for the processor.

Reading memory and load correctly

Two indicators are very often misread. First, memory: Linux uses free RAM as disk cache. A free value close to zero is therefore normal. The column that matters is available, the estimate of memory usable without resorting to swap.

Second, the load average: the three values shown by uptime are 1, 5 and 15-minute averages of the number of processes running or waiting to run. On Linux they also include processes blocked waiting for I/O. A load of 4 on a 4-core machine means the processor is fully busy; beyond that, tasks are waiting.

# Memory: look at the "available" column, not "free"
free -h

# Load average over 1, 5 and 15 minutes, and number of cores
uptime
nproc

# Summary view every second, 5 times:
# r = processes waiting for CPU, si/so = swap, wa = I/O wait
vmstat 1 5

# Resource pressure (kernel 4.20+)
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

In the vmstat output, an r column consistently above the number of cores indicates CPU saturation, and non-zero values in si/so mean the machine is actively swapping, which severely degrades performance. The /proc/pressure/* files (PSI) directly give the percentage of time during which tasks were delayed for lack of a resource.

Monitoring with Prometheus and Node Exporter

Interactive tools only show the present moment. To understand a gradual degradation or analyse an incident that happened overnight, you need historical metrics. The Prometheus and Node Exporter pair has become the standard: Node Exporter exposes system metrics (CPU, memory, disks, network, file systems) on port 9100, and Prometheus scrapes them at regular intervals.

# Install Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

Check the latest version on the project's releases page before installing. Then, rather than starting the binary by hand, run it as a systemd service under a dedicated user with no shell:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | grep node_load1

Listening on 127.0.0.1 avoids exposing the metrics to the Internet. If Prometheus runs on another machine, listen on the private interface and restrict access to port 9100 with the firewall. On the Prometheus side, you only need to add a target:

# prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 15s
    static_configs:
      - targets: ['10.0.0.5:9100']

System alerts

Set up alerts for critical metrics:

  • CPU: alert above 80% for 5 minutes
  • Memory: alert above 90%
  • Disk: alert above 85%
  • Load average: alert above the number of CPUs

Translated into Prometheus rules, these thresholds give the following file. The for clause requires the condition to stay true for a minimum duration, which avoids being woken up by a spike lasting a few seconds:

# /etc/prometheus/rules/node.yml
groups:
  - name: node
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: critical
      - alert: DiskAlmostFull
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85
        for: 10m
        labels:
          severity: warning
      - alert: HighLoadAverage
        expr: node_load5 > on (instance) count by (instance) (node_cpu_seconds_total{mode="idle"})
        for: 10m
        labels:
          severity: warning

The memory rule relies on MemAvailable rather than free memory, for the reasons explained above. The load rule compares node_load5 with the number of cores, counted from each instance's CPU series. Validate the file with promtool check rules /etc/prometheus/rules/node.yml before reloading Prometheus, then let Alertmanager handle sending notifications (email, Slack…).

Custom monitoring scripts

On a small standalone server without a Prometheus stack, a script run by cron can be enough to raise the alarm:

#!/bin/bash
# check_resources.sh
CPU=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}')
MEM=$(free -m | awk 'NR==2{printf "%.1f", $3*100/$2}')
DISK=$(df -h / | awk 'NR==2{print $5}' | tr -d '%')

echo "CPU: ${CPU}% | MEM: ${MEM}% | DISK: ${DISK}%"

if (( $(echo "$CPU > 80" | bc -l) )); then
  echo "ALERT: high CPU!" | mail -s "Server alert" admin@example.com
fi

This script has a few limitations worth knowing. The CPU value extracted from top is user time only (us), and the line format depends on the version and the locale. The computed memory is "used" memory, which excludes cache. Finally, the mail command assumes a mail agent is configured on the machine. Schedule it in the crontab:

*/5 * * * * /usr/local/bin/check_resources.sh >> /var/log/check_resources.log 2>&1

Common pitfalls

  • Too many alerts: an alert that fires every day with no action is quickly ignored. Every alert should map to a concrete action.
  • Only monitoring from the inside: a server can have excellent metrics and still be unreachable. Add an external probe on your public URLs.
  • Forgetting inodes: a disk can run out of inodes while still having free space, typically because of millions of small session or cache files. Check with df -i.
  • Ignoring the trend: a disk at 70% gaining 5% per day is more urgent than a disk stable at 88%. Prometheus's predict_linear function lets you alert on the predicted fill date.

In summary

For diagnosis, follow a method (utilisation, saturation, errors) and read available memory and load correctly. For the long run, install Node Exporter as a service, scrape it with Prometheus and define a small number of alerts with realistic thresholds and durations. The best monitoring is not the one that measures everything, but the one that warns you in time, and only when it matters.