Linux Disk I/O Monitoring: A Six-Symptom Decision Tree and Five Real Incidents
The Call That Started It All
The alert was boring: load average 12 on an 8-core box, and nothing else. I SSH'd in expecting a runaway process. Then I ran top and blinked.
%Cpu(s): 2.1 us, 1.4 sy, 0.0 ni, 36.2 id, 60.0 wa, 0.3 hi, 0.0 si
KiB Mem : 32783752 total, 1862112 free, 4102388 used, 26821252 buff/cache
KiB Swap: 4194300 total, 0 free, 4194300 used, 0 cached
PID USER %CPU %MEM TIME+ COMMAND
1234 mysql 0.3 1.2 12:44.03 mysqld
Sixty percent of the CPU was waiting, not working. The load was high, the cores were idle, and no process was to blame. This is the single most misread signal in Linux performance work: high load with idle CPU is almost always disk I/O wait, not CPU starvation. That night I learned to distrust load average as a headline and to go looking underneath it. This article is the field guide I wish I'd had.
vmstat: Three Numbers That Frame the Whole Problem
Before touching iostat, I take one vmstat snapshot. Three columns tell me which subsystem is lying.
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 4 4194296 1845232 210 26821252 0 0 4 210 102 1104 2 1 36 60 0
1 4 4194296 1845122 220 26821252 0 0 2 180 101 1088 2 1 35 61 0
1 4 4194296 1845012 230 26821252 0 0 5 190 102 1096 2 1 35 60 0
r— runnable processes. Low here means nobody is actually waiting for CPU.b— processes in uninterruptible sleep (D-state), the ones the load average is counting for free.wa— the iowait percentage; 60% here is the headline.
My rule of thumb: when r is small but b and wa are large, the load average is a disk problem wearing a CPU costume. When both r and wa are large, the disk is a symptom and the CPU is the disease.
The si/so columns are the next stop. Non-zero so with a full swap file is a second, sneakier disk killer, because every page written to swap is a page the disk has to absorb. More on that in the incidents.
The Six-Symptom Decision Tree
Over the years I've boiled every disk emergency down to six questions. I answer them in order and each one prunes the tree. I've drawn it below as the order I actually run the commands.
| Step | Symptom I see | What it usually means | Command |
|---|---|---|---|
| 1 | Load high, CPU idle, wa high |
Iowait — disk is the bottleneck | vmstat 1 |
| 2 | App slow but %util near 100% |
Could be fake saturation on NVMe | iostat -x 1 |
| 3 | Many D-state processes, nothing in r |
Blocked on I/O, queue is the wall | ps -eo state,wchan |
| 4 | Small random reads vs big sequential writes | Random access dominates seek time on HDD | iostat -x 1 r/s vs w/s |
| 5 | One process vs system-wide noise | Single culprit vs kernel thread (flush/jbd2) | iotop -oPa |
| 6 | kswapd0 or flush on top of iotop |
Writeback or memory pressure, not an app | pidstat -d 1 |
The point of the tree is to stop guessing. Symptom 1 gets me off the CPU and onto the disk. Symptom 2 stops me from trusting a number that lies. Symptoms 3–6 tell me who is hitting the disk and how.
iostat -x 1: Reading the Fields Like a Pro
iostat -x 1 is the single most informative command in the whole diagnosis, and the one people read most badly. The output is wide, so I trim it to the fields that matter.
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz await %util
sda 4.0 18.0 32.0 4096.0 0.0 0.0 0.0 0.0 0.20 24.80 0.45 8.0 227.0 20.80 42.0
The columns I read, in order:
%util— fraction of wall-clock time a request was in flight. On a spinning disk this is a solid saturation proxy. On NVMe it is not (incident #4). I never call a device saturated on%utilalone.await— average time from submission to completion, per I/O. This is the latency the application actually feels. Healthy NVMe: under 1 ms. Healthy SSD: under 2 ms. Healthy HDD: 5–15 ms.r_await/w_await— split of the above. Ifw_awaitis 10xr_await, writes are the problem, which points to writeback, journaling, or a synchronous-commit app.aqu-sz— average queue depth. Ifaqu-szis high butawaitis low, the device is capable and the queue is just buffering; the app is fine. Ifaqu-szis low andawaitis high, the device itself is slow.rrqm/s/wrqm/s— merges per second. A highwrqm/stells me the kernel is already merging adjacent writes well; a low one with a smallwareq-szsuggests a random-write pattern that defeats the elevator.
The two-number rule I quote in every postmortem: %util tells you the device is busy; await tells you it's slow. A device can be busy and fast (fine), or busy and slow (broken), or idle and slow (queueing elsewhere).
iotop and pidstat: From Disk to Process
Once I know the disk is the wall, I want the name on the wall. iotop gives me per-process I/O in real time, but the default output is noisy with cumulative counters.
Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.10 M/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 2.30 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
123 be/4 mysql 0.00 B 3.90 M 0.00 % 61.5 % mysqld
456 be/4 root 0.00 B 0.20 M 0.00 % 8.2 % kworker/u8:2-flush
The flags matter: -o shows only processes doing I/O right now, -P shows per-process (not per-thread) lines, -a accumulates. When a kernel thread like kworker/u8:2-flush or kswapd0 shows up near the top, that is the writeback path, not an application. That distinction alone has saved me from restarting the wrong service twice.
pidstat -d 1 is the non-interactive sibling, perfect for scripting and for catching short bursts.
12:00:01 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
12:00:02 27 123 0.00 4096.00 0.00 612 mysqld
iodelay is gold: milliseconds the process spent blocked waiting on I/O. A high iodelay with a low %CPU is the definitive "this process is disk-bound" signature.
D-State Processes: When the Kernel Can't Sleep
The load average counts processes in uninterruptible sleep. When the queue backs up, processes pile into D-state and the load number climbs while the CPU idles. ps shows them immediately.
I count D-state processes per command and per wait channel. wait_on_page_bit and __lock_page_killable are page-cache waits — the process is blocked on the disk doing the I/O. do_blockdev_direct_IO is direct I/O, which bypasses page cache and hits the device raw. A pile of D-state with a low r in vmstat is the textbook "everything is queued on the disk" picture, and it confirms symptom 3 of the tree.
The Drive Type Trap: HDD, SSD, NVMe Thresholds
The same iostat numbers mean different things on different hardware. This is the most dangerous part of disk monitoring, because the tool is identical while the physics is not. My matrix:
| Metric | HDD (spinning) | SSD (SATA) | NVMe |
|---|---|---|---|
await healthy |
5–15 ms | < 2 ms | < 1 ms |
await alarmed |
> 50 ms | > 20 ms | > 10 ms |
%util meaning |
True saturation proxy | Rough proxy | Nearly meaningless |
| Queue depth | Shallow (1–4) | Moderate | Deep (32+) |
| Sequential vs random | Huge gap | Small gap | Minimal gap |
On a spinning disk, %util near 100% genuinely means the head is always seeking and throughput has plateaued. On NVMe, the drive completes I/O in microseconds and pipelines thousands of requests; %util can sit at 100% while the application is perfectly happy (incident #4). For NVMe I trust await and aqu-sz almost exclusively, and I treat %util as decoration.
The seek penalty also explains why random I/O murders an HDD but barely dents an NVMe. A 4 KB random read on a 7200 RPM disk costs a seek (~8 ms) plus a rotation (~4 ms) — roughly 2000x the data-transfer time. The same read on NVMe is a single memory-like access. When I see a database doing thousands of small random reads on spinning media, the disk isn't slow, the workload is wrong for the hardware.
cgroup v2 blkio: Container-Level I/O Isolation
A large share of the disk incidents I handle now live inside containers, and the host-level tools only show the aggregate. cgroup v2 gives me the two knobs I reach for constantly.
# Cap a container to 20 MB/s and 100 IOPS
echo "8:0 rbps=20971520 wbps=20971520 riops=100 wiops=100" \
> /sys/fs/cgroup/io.max
io.max caps bandwidth (bps) and IOPS (iops) for read and write independently. This is the hammer for the noisy-neighbour problem: one container doing a bulk import was starving every API pod on the box until I capped it.
# Share the device 2:1 between two cgroups
echo "8:0 wbps=10485760" > /sys/fs/cgroup/group-a/io.max
echo "8:0 wbps=5242880" > /sys/fs/cgroup/group-b/io.max
io.weight (1–10000) is the proportional alternative: two cgroups at weight 200 and 100 get a 2:1 share of whatever the device can give, with no hard ceiling. The practical sequence is io.weight for fairness, and io.max when one tenant is actively misbehaving. I verify with the same iostat/iotop tools, read per-cgroup, which lets me prove the isolation actually worked.
Five Incidents I Actually Lived Through
Incident 1 — Backups that forgot to be lazy. A nightly rsync of a 2 TB directory was saturating the disk for hours. iotop -oPa showed rsync writing 90 MB/s while the web app crawled. The fix was throttling, and I did it at the source with --bwlimit rather than by fighting the OS:
Incident 2 — The invisible swap thrash. Load was 20, CPU was 30% idle, and no application was hot. iotop -oPa showed kswapd0 writing 40 MB/s of swap. vmstat had so non-zero. The app was leaking anonymous memory and the kernel was churning swap, not disk. The disk was a victim; the fix was memory, not I/O. I increased the memory limit and the iowait vanished.
Incident 3 — Database random I/O on spinning disks. A Postgres box on 7200 RPM disks had await at 90 ms and %util at 100%. iostat -x 1 showed 200 r/s of tiny 8 KB reads with a deep queue — classic random access. io.max could not fix physics. The real fix was storage, and the honest answer was to move to SSD; in the meantime, raising shared_buffers and forcing sequential prefetch cut the random reads substantially.
Incident 4 — The NVMe that looked saturated and wasn't. An NVMe box reported %util = 100% constantly, and the team was ready to replace it. But await was 0.4 ms and aqu-sz was 60. The drive was completing requests in half a millisecond; 100% busy just meant there was always at least one request in flight. The application was fine. We did nothing, and the right thing was to do nothing. This is the case that taught me to stop trusting %util on NVMe.
Incident 5 — ext4 journaling that doubled the writes. A write-heavy service had w_await at 30 ms while r_await was 1 ms. iotop showed a single process writing, and wrqm/s was low — few merges, meaning writes were being forced to disk. The filesystem was data=journal, which writes each transaction to the journal and the data blocks. Switching to data=ordered cut the write volume roughly in half, and w_await dropped to a healthy range.
FAQ
Q: Why is load average high when CPU usage is low?
A: Load counts both runnable and uninterruptible-sleep tasks. Processes blocked on I/O (D-state) inflate the number while consuming no CPU. A busy disk therefore reads as high load with idle cores; check vmstat b and wa before blaming the CPU.
Q: Is %util = 100% always saturation?
A: No. %util is the fraction of time a request was in flight, not throughput or latency. NVMe drives queue deeply and report 100% while completing I/O in under a millisecond. Always pair %util with await; on NVMe, trust await.
Q: What does await above 20 ms mean on an SSD?
A: A healthy SSD answers in single-digit milliseconds. await above 20 ms with a shallow aqu-sz means the device itself is slow (contention, firmware, or a failing backend), while a deep queue means the workload is outpacing it. Check w_await to split reads from writes.
Q: How do I find which process is hammering the disk?
A: Run iotop -oPa for live per-process I/O, or pidstat -d 1 for sampled rates and iodelay. If the top entry is kswapd0, flush, or jbd2, the cause is memory pressure or journaling rather than an application.
Q: How does cgroup v2 blkio help?
A: io.max caps a cgroup at a bandwidth or IOPS ceiling, and io.weight shares the device proportionally. Together they stop one noisy container from starving its neighbours, which I verify per-cgroup with iostat.