A crash does you at least one favor: it leaves behind a stack that looks guilty. Even if that stack points at the victim rather than the killer, it hands you a starting point, an alarm, a place to open the investigation.
A hang won’t even give you that. What it hands you is a process that is still running: memory flat, CPU anywhere from idle to fully pegged, every thread present, every thread waiting — and here is the maddening part — every single stack, examined on its own, is perfectly legal.
Let’s lay out the scene first. Twenty lines of code:
// deadlock.c - two locks, opposite order: every stack is innocent
#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
pthread_mutex_t m1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t m2 = PTHREAD_MUTEX_INITIALIZER;
void* worker_a(void* arg) {
(void)arg;
pthread_mutex_lock(&m1);
printf("[A] got m1\n");
sleep(1); // give B time to take m2, setting up the inversion
pthread_mutex_lock(&m2); // will wait forever: m2 is in B's hands
printf("[A] got both\n"); // unreachable
pthread_mutex_unlock(&m2);
pthread_mutex_unlock(&m1);
return NULL;
}
void* worker_b(void* arg) {
(void)arg;
sleep(1); // let A take m1 first
pthread_mutex_lock(&m2);
printf("[B] got m2\n");
pthread_mutex_lock(&m1); // will wait forever: m1 is in A's hands
printf("[B] got both\n"); // unreachable
pthread_mutex_unlock(&m1);
pthread_mutex_unlock(&m2);
return NULL;
}
void* worker_c(void* arg) {
(void)arg;
sleep(2); // innocent bystander: arrives late, just wants to borrow m1
pthread_mutex_lock(&m1); // unreachable: not a member of the cycle, a victim
printf("[C] got m1\n");
pthread_mutex_unlock(&m1);
return NULL;
}
int main(void) {
// Demo aid: Ubuntu's default ptrace_scope=1 forbids the debugger from
// attaching to sibling processes; this lets it in
prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY);
setvbuf(stdout, NULL, _IONBF, 0); // a hung program never flushes; kill buffering
printf("m1 @ %p\n", (void*)&m1);
printf("m2 @ %p\n", (void*)&m2);
pthread_t a, b, c;
pthread_create(&a, NULL, worker_a, NULL);
pthread_create(&b, NULL, worker_b, NULL);
pthread_create(&c, NULL, worker_c, NULL);
pthread_join(a, NULL);
pthread_join(b, NULL);
pthread_join(c, NULL);
printf("all done\n"); // unreachable
return 0;
}
Build and run it (gcc -g -O0 -pthread deadlock.c -o deadlock); the program prints its opening lines and then goes still. Following the instinct the crash era trained into you, grab all thread stacks and look for “where the problem is”:
$ gdb -p $(pidof deadlock) -batch -ex "thread apply all bt"
Thread 4 (Thread 0x74056f9ff6c0 (LWP 500) "deadlock"):
#0 futex_wait (private=0, expected=2, futex_word=0x5b3b71bd3080 <m2>) at ../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait (futex=futex@entry=0x5b3b71bd3080 <m2>, private=0) at ./nptl/lowlevellock.c:49
#2 lll_mutex_lock_optimized (mutex=0x5b3b71bd3080 <m2>) at ./nptl/pthread_mutex_lock.c:48
#3 ___pthread_mutex_lock (mutex=0x5b3b71bd3080 <m2>) at ./nptl/pthread_mutex_lock.c:93
#4 worker_a (arg=0x0) at deadlock.c:16
#5 start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#6 clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Thread 3 (Thread 0x74056f1fe6c0 (LWP 501) "deadlock"):
#0 futex_wait (private=0, expected=2, futex_word=0x5b3b71bd3040 <m1>) at ../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait (futex=futex@entry=0x5b3b71bd3040 <m1>, private=0) at ./nptl/lowlevellock.c:49
... (middle glibc frames identical to Thread 4; the awaited lock is m1)
#4 worker_b (arg=0x0) at deadlock.c:28
Thread 2 (Thread 0x74056e9fd6c0 (LWP 502) "deadlock"):
#0 futex_wait (private=0, expected=2, futex_word=0x5b3b71bd3040 <m1>) at ../sysdeps/nptl/futex-internal.h:146
... (same as above)
#4 worker_c (arg=0x0) at deadlock.c:38
Thread 1 (Thread 0x74056fc57740 (LWP 498) "deadlock"):
#3 __pthread_clockjoin_ex (...) at ./nptl/pthread_join_common.c:102
#4 main () at deadlock.c:54
Interrogate them one by one. Thread A sits inside pthread_mutex_lock, waiting for a lock — legal; everyone writes that. Thread B sits inside pthread_mutex_lock too — equally legal. Thread C merely wanted to borrow m1 for a moment — more innocent still. The main thread is in pthread_join, waiting for its children to come home, lawful beyond reproach. Not one stack contains an error. Not one line of code deserves suspicion.
The bug is real, but it doesn’t live inside any single stack. It lives in the relationships between stacks: A holds m1 and waits for m2; B holds m2 and waits for m1. That is a cycle in a graph — and “cycle” is a fact no single stack can express.
This article is the differential diagnosis of stillness. It picks up where The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging left off: the adversary there was “observation changes the system”; in heap-rot cases it is “the crash scene evaporates”; here it is the mirror image of both — a system that won’t even crash for you: no exception, no alarm, every state you read a legal state.
Every Windows-side output in this article was captured live on Windows 11 + MSVC 2026 + WinDbg (the cdb command line), and every Linux-side output on Ubuntu 24.04 + gcc 13.3 + gdb 15.1 + glibc 2.39; all programs are reproducible. Three practical gotchas, stepped on for you in advance: first, Ubuntu ships with yama.ptrace_scope=1, which forbids gdb from attaching to sibling processes — that is what the prctl(PR_SET_PTRACER) line in the demos is for; second, once a program hangs its stdout buffer never flushes again, so the demos call setvbuf to disable buffering — otherwise you can’t even see what happened before the incident; third, running TSan on WSL2 may die with unexpected memory mapping (the kernel’s high-entropy ASLR collides with the shadow-memory layout) — setarch $(uname -m) -R turns randomization off and fixes it.
1. “It’s Stuck” Is a Reading, Not a Diagnosis
The first sentence of most hang investigations is “the program stopped moving.” It gets treated as a statement about mechanism. It is actually a lab report.
You watched responsiveness: requests timed out, the progress bar froze, the log stopped scrolling. You picked a progress function — requests completed, log lines emitted, heartbeats — took its derivative over some window, and got zero. That is the entire content of “the program hung”: a progress function of your choosing, on a dimension of your choosing, returned zero.
The trouble is that behind this one zero reading hide at least five completely different mechanisms. Their causes sit at different positions on the time axis, they afflict different participants, and they demand exactly opposite forensic procedures:
| Type | The tense of the cause | The mechanism in one sentence |
|---|---|---|
| Deadlock | Structurally cancelled future | The wait-for graph has a cycle; what you wait for will never come |
| Lost wakeup | A past already spent | The wake fired before the wait began, and nobody registered it |
| Livelock | High-speed spinning in the present | Every instantaneous snapshot is legal; every unit of effort cancels the last |
| Starvation | Perpetual postponement in the present | The system as a whole progresses; one participant just never gets a turn |
| Phantom hang | Inside the observer’s expectations | Not stopped at all — slow, or alive on a dimension you aren’t watching |
Under livelock the CPU can be pegged. Under starvation the other threads race ahead. In a phantom hang the process may be deep in GC or waiting on the network — all zeros on your progress bar. Treating a reading as a diagnosis is where every wrong turn begins. “It hung” is like “he has a fever”: a symptom, not a disease.
There is a sneakier corollary, too. Crash investigations run “collect evidence first, reason later” — grab the dump, ask questions after. Hang investigations must run the other way around: reason first, collect evidence second. The five diseases impose orthogonal requirements on the scene. A deadlock’s scene is frozen; grab the dump whenever you like. A livelock’s scene is flowing; a single dump is waste. A lost wakeup’s scene is already dead; a hundred dumps will never photograph the cause. Starvation’s scene is not one observation but a distribution. A phantom hang’s scene lives on another dimension entirely; first you need a different yardstick. The forensic tool is a function of the diagnosis. Crash work teaches you to preserve the scene; this piece teaches you to ask, first, whether the scene is even worth preserving.
2. Triage: Three Cheap Readings That Sort the Families
The good news: triage needs no heavy machinery. Three layers of readings, each finer than the last, all from the operating system’s built-in observation surface.
Layer one: CPU. Task Manager, or top. A “hang” with pegged CPU and a “hang” with zero CPU are two different worlds: the former is either livelock or a busy-waiting variant of the phantom hang; the latter belongs to the waiting family. One line of output cuts the hypothesis space in half.
Layer two: the state letter. The STAT column of ps on Linux: R is running, S is sleeping (interruptibly), D is sleeping uninterruptibly — usually disk or network I/O, immune even to kill -9, the most deceptive form the phantom hang takes. The Windows equivalents are the thread states in Task Manager, or in the dump itself.
Layer three: the wait channel (wchan). Linux’s cheap gift to examiners: ps -eLo pid,tid,stat,wchan:24,comm tells you which kernel function each thread is sleeping in. This is triage’s microscope. Three real scenes, side by side (all output captured live):
# Scene A: deadlock
$ ps -eLo pid,tid,stat,wchan:24,comm | grep deadlock
PID TID STAT WCHAN COMMAND
498 498 Sl+ futex_do_wait deadlock
498 500 Sl+ futex_do_wait deadlock
498 501 Sl+ futex_do_wait deadlock
498 502 Sl+ futex_do_wait deadlock
# Scene B: phantom hang (blocked on I/O)
$ ps -eLo pid,tid,stat,wchan:24,comm | grep fakehang
PID TID STAT WCHAN COMMAND
543 543 Sl+ futex_do_wait fakehang
543 545 Sl+ anon_pipe_read fakehang
543 546 Sl+ hrtimer_nanosleep fakehang
# Scene C: starvation (livelock looks the same)
$ ps -eLo pid,tid,stat,wchan:24,comm | grep starvation
PID TID STAT WCHAN COMMAND
610 610 Sl+ hrtimer_nanosleep starvation
610 612 Rl+ - starvation
610 613 Rl+ - starvation
To the user, all three of these processes are “hung.” But in scene A all four threads sleep in futex_do_wait — the whole table is the wait-for graph in compressed form; everyone is waiting on a lock. In scene B the three threads sleep on three different channels (a lock, a pipe, a timer) — there is no cycle anywhere, just one thread waiting for input that will never arrive. In scene C both threads are in R state and don’t even have a wchan — they aren’t waiting at all; they’re running.
By this point the family membership of the five diseases is mostly settled, and what comes next is each family’s forensics and casework. One dissection at a time.
3. Deadlock: The Cause Lives in the Future, the Scene Freezes Forever
The Mechanism: A Cycle Is a Mathematical Fact, Indifferent to Time
The textbook definition of deadlock lists four conditions (mutual exclusion, hold-and-wait, no preemption, circular wait), but during an investigation only the last one matters: the wait-for graph contains a cycle. Treat each thread as a node, draw an edge for “thread X is waiting for a resource held by thread Y,” and the moment the graph closes a loop, every thread on that loop is permanently still.
One property here is worth chewing on: a cycle is a structural fact. It depends on no timing luck. Once it exists it will not heal while you wait, and it will not untangle under a different load. That is why a deadlock scene is frozen forever — not “nothing has happened yet” but “nothing will ever happen again.” For a forensic examiner this is rare good news: crash scenes evaporate (the entire anxiety of heap-rot work), while deadlock scenes stay fresh forever — autopsy them whenever you like.
The price: the cycle does not live inside any single stack. Go back to the gdb output from the introduction — four stacks, all legal. Closing the case means excavating the relationship, and the evidence sits in two places.
Evidence Source One: What Each Stack Waits On. Evidence Source Two: Who Each Lock Belongs To
The intro’s gdb output already hides half the graph: futex_wait (futex_word=0x5b3b71bd3080 <m2>) — gdb annotates which lock each thread is waiting on (the m1/m2 addresses printed at startup corroborate them). The other half is inside the locks themselves:
(gdb) p m1
$1 = {__data = {__lock = 2, __count = 0, __owner = 500, __nusers = 1, ...
(gdb) p m2
$2 = {__data = {__lock = 2, __count = 0, __owner = 501, __nusers = 1, ...
__owner is the holding thread’s ID (the LWP) stored inside the glibc mutex. Cross-reference info threads: LWP 500 is worker_a, LWP 501 is worker_b. Put the two halves together and the graph closes into a cycle:
worker_a (LWP 500) --holds m1, waits for m2--> m2's owner is LWP 501 (worker_b)
worker_b (LWP 501) --holds m2, waits for m1--> m1's owner is LWP 500 (worker_a)
worker_c (LWP 502) --waits for m1--> a victim queued at the door of the scene
main (LWP 498) --join--> likewise
Note the last two lines. Your dump contains four “hung” stacks, but only two of them are culprits; the other two are victims. In a real service the ratio runs the other way: two threads on the cycle, a hundred victims queued outside the door (the entire thread pool stacked up in line). Treating victims as culprits is mistake number one in deadlock work — the exact trap of “what crashes is the victim” in heap rot, except there the lie was one stack and here it’s ninety-eight.
The Windows Version: !cs Turns the Evidence Chain into One Command
The same play is staged on Windows, with CRITICAL_SECTION as the prop. After the program (cs_deadlock.cpp, a line-for-line twin of the Linux version) hangs, attach cdb:
0:004> ~*k
0 Id: 69f0.6e74
ntdll!NtWaitForMultipleObjects+0x14
KERNELBASE!WaitForMultipleObjectsEx+0x123
KERNELBASE!WaitForMultipleObjects+0x11
cs_deadlock!main+0xe2 [D:\tmp\hang\cs_deadlock.cpp @ 46] <- main waiting on its three children
1 Id: 69f0.3710
ntdll!NtWaitForAlertByThreadId+0x14
ntdll!RtlpWaitOnCriticalSection+0x5ad
ntdll!RtlpEnterCriticalSectionContended+0x1ef
ntdll!RtlEnterCriticalSection+0xf2
cs_deadlock!worker_a+0x3a [D:\tmp\hang\cs_deadlock.cpp @ 13] <- waiting on the second lock
2 Id: 69f0.6f80
... (structurally identical to thread 1)
cs_deadlock!worker_b+0x3a [D:\tmp\hang\cs_deadlock.cpp @ 24]
3 Id: 69f0.6a08
... (likewise -- the innocent bystander)
cs_deadlock!worker_c+0x21 [D:\tmp\hang\cs_deadlock.cpp @ 33]
The stacks only tell you who is waiting. Who is holding? Enter !cs (this needs type-complete ntdll symbols from a symbol server — export tables alone aren’t enough):
0:004> !cs -l
-----------------------------------------
Critical section = 0x00007ff68467d1d8 (cs_deadlock!g_cs2+0x0)
LOCKED
LockCount = 0x1
OwningThread = 0x0000000000006f80 <- = thread 2's TID (worker_b)
RecursionCount = 0x1
-----------------------------------------
Critical section = 0x00007ff68467d1b0 (cs_deadlock!g_cs1+0x0)
LOCKED
LockCount = 0x2
OwningThread = 0x0000000000003710 <- = thread 1's TID (worker_a)
RecursionCount = 0x1
OwningThread matched against the thread numbers in ~*k, and the cycle closes. The two LockCount values (2 and 1) happen to equal the number of waiters at each door: worker_b and worker_c are queued at g_cs1, worker_a alone at g_cs2.
And then there is the trump card: !cs -o prints the owner’s stack directly:
0:004> !cs -o cs_deadlock!g_cs2
-----------------------------------------
Critical section = 0x00007ff68467d1d8 (cs_deadlock!g_cs2+0x0)
LOCKED
OwningThread = 0x0000000000006f80
OwningThread Stack =
ntdll!NtWaitForAlertByThreadId+0x14
ntdll!RtlpWaitOnCriticalSection+0x5ad
ntdll!RtlpEnterCriticalSectionContended+0x1ef
ntdll!RtlEnterCriticalSection+0xf2
cs_deadlock!worker_b+0x3a
Read that conversation back. g_cs2 says “my owner is 6f80”; 6f80’s stack says “I’m stuck in RtlEnterCriticalSection.” Entering which lock? The RtlpEnterCriticalSectionContended frame carries 0x00007ff68467d1b0 in its arguments — g_cs1’s address. The lock gave up its owner; the owner’s stack gave up the next lock it wants; that lock points back at the first. All three edges of the cycle, verified by a single command.
Invisible Locks: The Locks You Never Wrote Are the Most Dangerous
In the two cases above, the locks on the cycle were your own, and the stacks at least contained visible signals like EnterCriticalSection or pthread_mutex_lock. The most deceptive deadlocks put not one lock you wrote on the cycle. The classic of classics: create a thread inside DllMain and wait for it to finish.
// ll_dll.cpp - create a thread in DllMain and wait for it: dying on an invisible lock
#include <windows.h>
#include <stdio.h>
static DWORD WINAPI work(void*) {
printf("[thread] new thread alive\n"); // unreachable: the new thread never starts
return 0;
}
BOOL WINAPI DllMain(HINSTANCE h, DWORD reason, LPVOID) {
if (reason == DLL_PROCESS_ATTACH) {
HANDLE t = CreateThread(0, 0, work, 0, 0, 0);
WaitForSingleObject(t, INFINITE); // main holds the loader lock, waits for the thread
CloseHandle(t);
}
return TRUE;
}
The host program calls LoadLibraryA("ll_dll.dll") and the whole process hangs. Grab all thread stacks:
0:002> ~*k
0 Id: 618c.6d28
ntdll!NtWaitForSingleObject+0x14
KERNELBASE!WaitForSingleObjectEx+0xaf
ll_dll!DllMain+0x53 <- main: waiting on the thread, in your code
ll_dll!dllmain_dispatch+0x96
ntdll!LdrpCallInitRoutineInternal+0x22
ntdll!LdrpCallInitRoutine+0x93
ntdll!LdrpInitializeNode+0x19c
ntdll!LdrpInitializeGraphRecurse+0x6a <- but simultaneously inside loader initialization
1 Id: 618c.5890
ntdll!NtWaitForSingleObject+0x14
ntdll!LdrpDrainWorkQueue+0x199
ntdll!LdrpInitializeThread+0xef <- the new thread: never starts; stuck in loader init
ntdll!LdrpInitialize+0xa7
ntdll!LdrpInitializeInternal+0x5a
ntdll!LdrInitializeThunk+0xe
No lock of yours appears anywhere on these stacks; both threads look like they’re waiting on the most innocent things in the world (one on a thread handle, one “initializing”). The evidence of the cycle lives in the semantics of the stacks: the main thread’s stack passes through loader-initialization frames (the LdrpInitializeGraphRecurse family) — which means it holds the loader lock; the new thread’s stack sits inside LdrpInitializeThread — which means it needs the loader lock (a new thread must broadcast THREAD_ATTACH to every loaded DLL, and the broadcast takes the loader lock). Autopsy the invisible lock:
0:002> !cs ntdll!LdrpLoaderLock
-----------------------------------------
Critical section = 0x00007ffcd522c898 (ntdll!LdrpLoaderLock+0x0)
LOCKED
OwningThread = 0x0000000000006d28 <- = the main thread (618c.6d28)
RecursionCount = 0x1
OwningThread = 0x6d28 — the main thread, the one stuck in DllMain. The cycle closes: the main thread holds the loader lock and waits for the new thread; the new thread waits for the loader lock. (Two version details worth pocketing: in modern ntdll, LdrpLoaderLock is the critical-section body itself, not a pointer — no poi() dereference needed; and !cs -l on Windows 11 lists only the few sections that still carry DebugInfo, so don’t expect a complete inventory from it.)
Invisible locks are the most treacherous branch of the deadlock family, and they share exactly one trait: the lock appears nowhere in your code — only on somebody else’s stack. Your code never calls an acquisition, yet the cycle exists all the same. Appendix C lists the common invisible locks. The diagnostic rule was demonstrated a moment ago: look at whether a stack passes through lock-holding code paths (loader initialization, allocator internals, CRT startup), not for lock-function names.
Deadlock Misdiagnosis Traps, Summarized
- Treating the queued victims outside the cycle as culprits (ninety-eight of the hundred stacks are people standing in line).
- Looking at stacks one at a time (a cycle is a relation; a single point cannot contain one).
- Concluding “no deadlock” because no lock-like function appears (judge by the paths the stacks pass through, not by function names — that’s where invisible locks hide).
- The one bad habit the frozen scene encourages: “I can grab a dump anytime, so let me do something else first” — production processes get restarted by ops at any moment, and forever-frozen does not mean forever-there. Triage first, then take your evidence immediately.
4. Livelock: The Cause Lives Between Snapshots, the Scene Keeps Moving
Deadlock is the system saying “I’ve stopped.” Livelock is the system telling you “I’m trying” — and every try cancels the last one. Two threads each want both locks, and both are unfailingly polite: can’t get the second? Release the first, back off, start over. If they stay forever half a beat out of phase, they will never hold both. CPU burns at full tilt; no work gets done.
To make “forever half a beat out of phase” deterministic, the demo uses a two-thread barrier (pthread_barrier_t) guaranteeing that each round, both threads hold their first lock “simultaneously” before both reach for the other’s; real-world livelock needs no such scaffolding — two flows of execution will find their adverse phase all on their own:
// livelock.c - retry storm: each holds one lock, both hammer at the other's
#define _GNU_SOURCE
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
pthread_mutex_t m1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t m2 = PTHREAD_MUTEX_INITIALIZER;
pthread_barrier_t bar; // two-thread barrier: keeps them each holding one, in lockstep
atomic_long attempts = 0; // retry count
atomic_long progress = 0; // work actually accomplished
enum { RETRIES = 10000 }; // attempts per round
void* worker(void* arg) {
int id = *(int*)arg;
pthread_mutex_t* first = (id == 0) ? &m1 : &m2; // A takes m1 first, B takes m2 first
pthread_mutex_t* second = (id == 0) ? &m2 : &m1;
for (;;) {
pthread_mutex_lock(first); // I hold my first lock
pthread_barrier_wait(&bar); // wait until the other holds his first too
// Now it is certain: second is in the other's hands. Retrying is futile --
// that is livelock's "effort": not stillness, but every unit of effort
// canceling the last one
for (int i = 0; i < RETRIES; ++i) {
atomic_fetch_add(&attempts, 1);
if (pthread_mutex_trylock(second) == 0) { // EBUSY, forever
atomic_fetch_add(&progress, 1);
pthread_mutex_unlock(second);
break;
}
}
pthread_barrier_wait(&bar); // wait until the other has tried too, then both let go
pthread_mutex_unlock(first); // release, and immediately start over
}
return NULL;
}
int main(void) {
prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY); // let the debugger attach
pthread_barrier_init(&bar, NULL, 2);
int ids[2] = {0, 1};
pthread_t t[2];
pthread_create(&t[0], NULL, worker, &ids[0]);
pthread_create(&t[1], NULL, worker, &ids[1]);
for (int s = 1; s <= 5; ++s) {
sleep(1);
printf("t+%ds : attempts = %-14ld progress = %ld\n",
s, atomic_load(&attempts), atomic_load(&progress));
}
return 0;
}
First, the triage reading (layer one, CPU):
$ top -H -p $(pidof livelock)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
413 feng-yr 20 0 19080 1752 1636 R 81.8 0.0 0:01.05 livelock
414 feng-yr 20 0 19080 1752 1636 R 81.8 0.0 0:01.04 livelock
411 feng-yr 20 0 19080 1752 1636 S 0.0 0.0 0:00.00 livelock
Two worker threads in R state, each burning most of a core. The five-second ledger:
t+1s : attempts = 25792070 progress = 0
t+2s : attempts = 43455439 progress = 0
t+3s : attempts = 57866690 progress = 0
t+4s : attempts = 73520000 progress = 0
t+5s : attempts = 100716754 progress = 0
One hundred million attempts in five seconds, zero successes. “Effort,” made concrete.
Now the critical comparison. With a deadlock you can grab one dump and put your feet up. With a livelock, grab three, half a second apart (captured live, all three from the same process):
$ gdb -p <pid> -batch -ex "thread apply all bt 4" -ex "p attempts" -ex "p progress"
--- snapshot 1 ---
Thread 3: #0 worker (...) at livelock.c:29 <- A, inside the retry loop
Thread 2: #0 ___pthread_mutex_trylock (mutex=0x5bbf5719c0c0 <m1>) <- B, deeper inside trylock
$1 = 35406813
$2 = 0
--- snapshot 2 ---
Thread 3: #0 ___pthread_mutex_trylock (mutex=0x5bbf5719c080 <m2>) <- A, in a different spot
Thread 2: #0 futex_wait (expected=9986, futex_word=0x5bbf5719c064 <bar+4>)
#2 ___pthread_barrier_wait (barrier=0x5bbf5719c060 <bar>) <- B, napping at the barrier
$1 = 49936699
$2 = 0
--- snapshot 3 ---
Thread 3: #0 worker (...) at livelock.c:29
Thread 2: #0 futex_wait (expected=12334, futex_word=0x5bbf5719c064 <bar+4>)
$1 = 61676615
$2 = 0
Three details deserve a close look. First, the stacks move: across snapshots the threads land in different places (bouncing between the retry loop, the inside of trylock, and the barrier wait). Second, attempts grew by more than ten million between snapshots — that counter is the pulse of the flowing scene. Third, the prettiest detail of all: the barrier futex’s expected value climbed from 9986 to 12334. It is a sequence number, incremented once per barrier round — all by itself, it is telling you the scene is in motion. Contrast that with deadlock: capture a hundred dumps of that program and the stacks, the __owner values, and the futex’s expected value will not change by a single character.
That is the full meaning of a scene’s “freezability”: a deadlock’s scene is a photograph; a livelock’s scene is a film. Taking photographs of a film is not forensics, it’s waste — in any single dump, one thread retries while another waits, indistinguishable from a healthy program. The correct forensics for livelock is sampling and counting: a profile from perf or ETW (the hot functions will be trylock and the spin loop), and the ratio between retry counters and success counters. One sentence: the difference between snapshots is the livelock’s scene.
(Real-world livelocks look far more respectable than this demo: two servers detect the same conflict and restart in unison, two consumers keep bouncing a failed message back and forth, two format converters keep translating a document into each other’s arms — wherever retry logic collides with retry logic, that is its habitat.)
5. Lost Wakeup: The Cause Lives in the Past, the Scene Is Already Dead
Deadlock’s cause lives in the future (what you’re waiting for will never arrive); livelock’s lives in the present (it’s spinning right now). The third kind of stillness is sneakier: the cause lives in the past, and it has already been spent.
A condition-variable wake is not a letter mailed to the future. It is a shout aimed at whoever is in the room at that moment. You were asleep when they shouted, and the shout is gone forever — a condition variable keeps no queue of “signals nobody was around for.” Unless, that is, the ledger recorded that the event happened.
// lostwakeup.c - a wake is a historical event: the shout landed in an empty room
#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cv = PTHREAD_COND_INITIALIZER;
int ready = 0; // the ledger: has the event happened yet
void* producer(void* arg) {
(void)arg;
sleep(1); // runs first: the event happens at t=1
pthread_mutex_lock(&m);
ready = 1; // book it
pthread_cond_signal(&cv); // "wake up!" -- shouted into an empty room
pthread_mutex_unlock(&m);
printf("[producer] event published, signal fired to empty room\n");
return NULL;
}
void* consumer(void* arg) {
(void)arg;
sleep(2); // arrives late: t=2
pthread_mutex_lock(&m);
// The lesion: never reads the ledger, just sleeps. The wake finished
// happening at t=1 and left no queue entry behind
#ifdef FIXED
while (!ready) pthread_cond_wait(&cv, &m);
#else
pthread_cond_wait(&cv, &m);
#endif
pthread_mutex_unlock(&m);
printf("[consumer] woke up, working\n"); // unreachable
return NULL;
}
The program prints one line, [producer] event published, signal fired to empty room, and then goes still. Grab the scene:
$ gdb -p $(pidof lostwakeup) -batch -ex "thread apply all bt" -ex "p ready" -ex "p cv"
Thread 2 (Thread 0x76b9c77fe6c0 (LWP 524) "lostwakeup"):
#0 __futex_abstimed_wait_common64 (futex_word=0x57dd1e2ed0a8 <cv+40>) at ./nptl/futex-internal.c:57
#3 __pthread_cond_wait_common (mutex=0x57dd1e2ed040 <m>, cond=0x57dd1e2ed080 <cv>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x57dd1e2ed080 <cv>, mutex=0x57dd1e2ed040 <m>) at ./nptl/pthread_cond_wait.c:627
#5 consumer (arg=0x0) at lostwakeup.c:33
...
(gdb) p ready
$1 = 1
(gdb) p cv
$2 = {__data = {__wseq = {__value64 = 2, ...
Read the autopsy report. The stack captured only “the posture of waiting” (pthread_cond_wait, asleep on the cv’s internal futex) — indistinguishable from a perfectly healthy wait. What actually names the disease is the ledger: ready = 1. The event happened; the ledger remembers it clearly; the sleeping thread never once looked. Even the condition variable’s internal counter __wseq = 2 (glibc numbers wait and signal events with it: one shout at t=1 plus one lie-down at t=2, two entries) left a trace — though that is an internal structure whose semantics nobody promises you, so you can’t build business logic on it, but it proves the shout happened.
This is what makes lost wakeup more thorough than a crash. A crash at least leaves a body and a time of death; a lost wakeup leaves only the posture of waiting, the cause being a historical event registered in no data structure. Capture a hundred dumps and you will get this same posture a hundred times. The information content of a dump, for this disease, is zero.
Two things can save you. The first is the ledger (the predicate): if the waiter reads the books before lying down — ready == 1 — it never sleeps at all. That is the entire content of the while rule:
Waking up never proves the event happened; only the ledger does. You can wake from a spurious wakeup, from someone else’s event, or for no reason at all — so on waking you must read the books again, and if it isn’t your turn yet, go back to sleep.
whileis not a style preference; it is a correctness boundary —iftreats “I woke up” as evidence that “the event happened,” an inference that does not hold in a concurrent world.
And the self-healing check, for free: compile with the while line enabled (gcc -DFIXED lostwakeup.c -o fixed), and the same program recovers instantly:
$ ./lostwakeup_fixed
[producer] event published, signal fired to empty room
[consumer] woke up, working
all done
The second thing that can save you is a time machine: record-and-replay. WinDbg TTD or rr records execution and plays it backwards, letting you watch with your own eyes “the t=1 signal land in an empty room and the t=2 wait arrive too late.” Lost wakeup is reverse debugging’s perfect use case, because its cause is a one-shot historical event — it doesn’t recur, doesn’t linger, leaves no scene. Only the tape can play it back.
6. Starvation: Unfairness Lives in the Distribution, the Scene Is a Statistic
In the fourth kind of stillness, the system as a whole is making healthy progress; exactly one participant is being postponed forever. What makes it devious is this: in any single snapshot, the starving thread looks identical to the well-fed one.
// starvation.c - starvation lives in the distribution: any single snapshot is innocent
#define _GNU_SOURCE
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
atomic_int tas = 0; // bare TAS lock: winner takes all, no promise of fairness
void lock_tas(void) {
while (atomic_exchange_explicit(&tas, 1, memory_order_relaxed)) { }
}
void unlock_tas(void) {
atomic_store_explicit(&tas, 0, memory_order_relaxed);
}
atomic_long wins_hog = 0; // the fast one: re-grabs immediately (zero gap)
atomic_long wins_slow = 0; // the slow one: steps out on errands, then comes back
void* hog(void* arg) {
(void)arg;
for (;;) {
lock_tas();
atomic_fetch_add(&wins_hog, 1);
unlock_tas();
}
return NULL;
}
void* slow(void* arg) {
(void)arg;
for (;;) {
lock_tas();
atomic_fetch_add(&wins_slow, 1);
unlock_tas();
for (volatile int i = 0; i < 300; ++i) { } // run a small errand outside
}
return NULL;
}
int main(void) {
prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY); // let the debugger attach
pthread_t t1, t2;
pthread_create(&t1, NULL, hog, NULL);
pthread_create(&t2, NULL, slow, NULL);
sleep(1);
printf("t+1s : wins_hog = %-12ld wins_slow = %ld\n",
atomic_load(&wins_hog), atomic_load(&wins_slow));
sleep(1);
printf("t+2s : wins_hog = %-12ld wins_slow = %ld\n",
atomic_load(&wins_hog), atomic_load(&wins_slow));
return 0;
}
Attach gdb midway through the run (captured live):
Thread 3: #0 hog (arg=<optimized out>) at starvation.c:26 <- doing work; the picture of health
Thread 2: #0 lock_tas () at starvation.c:12 <- grabbing the lock; equally healthy
#1 slow (arg=<optimized out>) at starvation.c:34
Inside one frame, nobody is sick. The disease lives in the allocation between frames:
t+1s : wins_hog = 45790715 wins_slow = 1395787
t+2s : wins_hog = 100211313 wins_slow = 3054837
Thirty-three to one. The starving thread is not “stuck” — it still wins over a million times per second; it even looks busy. But its relative share trends toward zero and never catches up. Starvation does not exist in any single observation; it exists only in the distribution of observations: mean latency looks normal (averaged away by the saturated thread), instantaneous state looks normal (it’s running), a one-shot dump looks normal. The only things that can name it are long-window statistics: per-thread acquisition counts, wait histograms, tail latency.
One honest footnote: glibc’s pthread_mutex is actually quite fair (the futex wait queue is roughly FIFO), which is why the demo uses a bare TAS lock — the thread that just unlocked still has the cache line hot in its hands, so it turns around and re-grabs immediately, and the slower contender stays half a step behind forever. Starvation lives where nobody has promised fairness: spinlocks, schedulers without priority inheritance, the polling order of message queues, the task-stealing boundaries of thread pools. The word you didn’t find in your lock’s documentation is the reason you’ll be crawling out of bed at 3 a.m. someday.
7. The Phantom Hang: The Cause Lives in Your Expectations, the Scene Is on Another Dimension
The first four diseases are diseases of the system. The fifth is a disease of the observer: the system isn’t dead — your yardstick just can’t reach the place where it’s alive.
// fakehang.c - phantom hang: not dead, just legally waiting for input that never comes
#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
int fd[2]; // a pipe nobody will ever write to
void* reader(void* arg) {
(void)arg;
char buf[16];
printf("[reader] about to read ...\n");
ssize_t n = read(fd[0], buf, sizeof buf); // legal, healthy, indefinite waiting
printf("[reader] got %zd bytes\n", n);
return NULL;
}
void* heartbeat(void* arg) {
(void)arg;
for (;;) {
putchar('.'); fflush(stdout);
usleep(300000);
}
return NULL;
}
int main(void) {
prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY);
pipe(fd);
pthread_t r, h;
pthread_create(&r, NULL, reader, NULL);
pthread_create(&h, NULL, heartbeat, NULL);
pthread_join(r, NULL);
printf("all done\n"); // unreachable
return 0;
}
To the user this program has “hung”: the reader thread does no work. But look at the output — the heartbeat is beating:
[reader] about to read ...
.......
And look back at scene B in section 2’s table: three threads, three wait channels (futex_do_wait / anon_pipe_read / hrtimer_nanosleep) — the process is alive and well; it’s just that on the dimension you chose to measure (bytes read) there are no events. Grab a dump and you’ll see the reader sleeping legally inside the read syscall, fd=3 — diagnostic value zero, because there is no pathology in this scene. The pathology is in your expectations: you assumed somebody would write to that pipe.
The phantom hang is a family, not a single disease: blocked on slow I/O (a cold disk cache, an NFS timeout, antivirus scanning the file you just opened), GC pauses, page-fault storms, waiting on an upstream response that will never arrive. Its two sub-species separate cleanly with section 2’s three readings: zero CPU (asleep on I/O, S or D state) versus pegged CPU (busy on another dimension — GC, or a spin-waiting variant). The D state is the most deceptive of all — uninterruptible disk wait, immune even to kill -9, which looks exactly like a deadlock’s unkillability — but it isn’t a cycle; it’s a yardstick problem.
The first corrective action for a phantom hang is not to grab a dump; it’s to change yardsticks: CPU, disk I/O, network, GC time, paging — measure every dimension once, and usually the problem introduces itself. It is also the only one of the five diseases deeply coupled with environment differences: when a “hang” vanishes on a different machine, odds are it was a phantom, because slow-versus-fast is an environment variable to begin with. Antivirus is only installed on the production image; the NFS mount is only reachable from the office network; the cold cache only exists right after a reboot: much of what you measured as “hung” was the environment, not the code.
8. A Unified Theory: Translating “Not Moving” into the Language of Order
Five diseases, five flavors of zero progress — it looks like a messy checklist. But all five can be rewritten in a single language: order.
| Type | The ordering pathology |
|---|---|
| Deadlock | A cycle in the wait-for partial order (a structural contradiction: A before B and B before A) |
| Lost wakeup | The wake completed its ordering before the wait began (a historical contradiction: the required relation was never established) |
| Livelock | The execution sequence extends forever while the progress predicate never holds (a liveness contradiction) |
| Starvation | The allocation order keeps favoring others (a fairness contradiction) |
| Phantom hang | Progress events exist, just not on the observed dimension (an observational contradiction: the first four are diseases of the system, this one is a disease of the observer) |
This viewpoint immediately pays an engineering dividend: some ordering contradictions can be proven without ever firing. The deadlock criterion is “the lock-order graph has a cycle,” and that graph can be maintained incrementally on every successful acquisition — you don’t need to actually die; you only need to observe the contradictory order twice. That is exactly what ThreadSanitizer’s deadlock detector does:
// deadlock_tsan.c - TSan's deadlock verdict: no need to actually die,
// only to contradict the order
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
pthread_mutex_t m1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t m2 = PTHREAD_MUTEX_INITIALIZER;
void* worker_a(void* arg) {
(void)arg;
pthread_mutex_lock(&m1);
pthread_mutex_lock(&m2); // ordering evidence #1: m1 -> m2
printf("[A] m1 -> m2 OK\n");
pthread_mutex_unlock(&m2);
pthread_mutex_unlock(&m1);
return NULL;
}
void* worker_b(void* arg) {
(void)arg;
sleep(1); // let A's order enter the graph first
pthread_mutex_lock(&m2);
printf("[B] hold m2, want m1\n");
pthread_mutex_lock(&m1); // ordering evidence #2: m2 -> m1 (nobody holds m1
// right now, so it succeeds)
printf("[B] m2 -> m1 OK\n");
pthread_mutex_unlock(&m1);
pthread_mutex_unlock(&m2);
return NULL;
}
int main(void) {
pthread_t a, b;
pthread_create(&a, NULL, worker_a, NULL);
pthread_create(&b, NULL, worker_b, NULL);
pthread_join(a, NULL);
pthread_join(b, NULL);
printf("all done\n"); // never deadlocks once -- but the verdict is already written
return 0;
}
Build and run (gcc -O1 -g -fsanitize=thread -pthread deadlock_tsan.c -o deadlock_tsan; on WSL2, remember to wrap it in setarch $(uname -m) -R):
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=464)
Cycle in lock order graph: M0 (0x555555558080) => M1 (0x555555558040) => M0
Mutex M1 acquired here while holding mutex M0 in thread T1:
#0 pthread_mutex_lock ... (libtsan.so.2+0x59a13)
#1 worker_a /mnt/d/tmp/hang/deadlock_tsan.c:12 (deadlock_tsan+0x12ba)
Mutex M0 acquired here while holding mutex M1 in thread T2:
#0 pthread_mutex_lock ... (libtsan.so.2+0x59a13)
#1 worker_b /mnt/d/tmp/hang/deadlock_tsan.c:24 (deadlock_tsan+0x132f)
...
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /mnt/d/tmp/hang/deadlock_tsan.c:12 in worker_a
==================
[A] m1 -> m2 OK
[B] hold m2, want m1
[B] m2 -> m1 OK
all done
Notice the last four lines: the program ran to completion — all done. It never deadlocked once, and yet the verdict was already written: the ordering rule for these two locks contradicts itself, the cycle has closed in the lock-order graph, and all that remains is the right interleaving to turn it into a physical accident. This is exactly the core of Non-Invasive Probes: How ASan/TSan/UBSan Expose Bugs Before They Ever Misbehave paying off here: TSan’s verdict term is the cycle in the lock-order graph (a logical fact); its perturbation term is timing — and timing is not in the verdict formula. It proves “exists” without waiting for “happens.”
One level up, and you arrive at this article’s place in the series. The happens-before partial order TSan uses to adjudicate data races is the ordering language Lamport invented in 1978 for distributed systems; deadlock’s cycles, lost wakeup’s “the wake preceded the wait,” starvation’s allocation bias — all of it can be stated in that language. That is no coincidence: a shared-memory multithreaded program is, logically, a distributed system — no privileged observer, every thread with its own viewpoint and its own causal history, no globally consistent “now.” Non-Invasive Probes planted this hook in its closing lines; this piece has lowered it onto the ground of stillness. The answer lives in some future article: when your system crosses processes and then machines, this craft of differential diagnosis will follow you over, unchanged.
9. Case Study: One “Every Request Times Out,” Two Different Diseases
Now assemble the parts into a full investigation. The case is the production-scale version of section 3’s experiment.
Background: a service with hot-reloadable configuration. On the request path, worker threads take stats_lock to update counters and occasionally read config (taking cfg_lock). On the reload path, the reload thread takes cfg_lock to swap in new config and then, as a courtesy, clears the statistics (taking stats_lock). Two locks, two paths, opposite orders. Three months in production without incident — until one traffic peak, when a reload happened to collide with a batch of in-flight requests: every request timed out, monitoring a sea of red.
The wrong investigation: the on-call engineer’s first suspect is the network (“timeouts, obviously”). Gateway logs, pings to upstream, connection-pool gauges — all normal. Half an hour burned before anyone looks at the service itself.
The correct investigation, four steps:
Step 1, triage. CPU near zero, every thread in S state, the wchan table a solid field of futex_do_wait, the request queue growing and never draining. Reading classified: stillness family, global zero progress, everyone waiting on locks — deadlock or lost wakeup, and you test for deadlock first (cheap forensics, frozen scene).
Step 2, preserve. Grab a full-thread dump (the scene is frozen forever, so no need to panic — but grab it now anyway: production processes get restarted by ops at any moment, and forever-frozen does not protect against external force). A hundred-odd stacks; ninety percent of them asleep at stats_lock‘s door.
Step 3, assemble the graph. Extract the wait edges from the stacks (each stack’s pthread_mutex_lock argument) and the hold edges from the locks themselves (__owner, or !cs‘s OwningThread), and draw the wait-for graph. The cycle surfaces: a worker holds stats_lock and waits for cfg_lock; reload holds cfg_lock and waits for stats_lock. Only two stacks sit on the cycle; the other hundred are queued victims — which is why “staring at the most-stuck stack” never finds the culprit.
Step 4, close and regress. The fix is a unified lock order: cfg_lock is always taken before stats_lock (or merge them into one lock, or take both at once with std::scoped_lock). And the regression is not “run it again and hope” — replay both paths under a TSan build; the lock-order-inversion report going from present to absent is what actually closes the case.
Variation (three weeks later): the same symptoms return. The readings are identical, so the team, now practiced, grabs a dump and assembles the graph — and the cycle doesn’t close. Everyone is legally waiting; the wait-for graph is a tree, not a loop. Ten dumps, ten trees. The investigation hangs on the hang.
This time you go back and read the ledger: the reload-complete flag reload_done is 1 — the event already happened — but the workers are in an unconditional pthread_cond_wait. Lost wakeup: the broadcast at reload time landed in an empty room, and the workers came to sleep later. The fix is while (!reload_done) wait. The retrospective is where the chill lives: why did those dumps carry zero information? Because the cause was in the past (the millisecond of the broadcast), and a dump can only photograph the present. Had the ledger not kept that entry, the only remaining forensics would have been to set up record-and-replay and watch what happened between “reload completed” and “worker lay down.”
Two faces, one reading. The first time, the cause was in the future (waiting for something that would never come); the second time, it was in the past (having missed something that already came). The first time, the frozen scene was a gift; the second time, the absent scene was inevitable. On the triage sheet they are neighbors; in the forensics kit they are separated by a time machine.
10. Method and Prevention: Reason First, Immunize Early
The four-step method, condensed to one table:
| Step | Action | The essential discipline |
|---|---|---|
| 1 Triage | CPU / state letter / wchan — three layers of readings | Settle the family before forensics; “it hung” is a reading, not a diagnosis |
| 2 Preserve | Pick the tool by family: frozen -> dump; flowing -> sampling; historical -> recording; distributional -> statistics; wrong dimension -> different yardstick | The forensic tool is a function of the diagnosis |
| 3 Assemble the graph | Wait edges (stacks) + hold edges (owner fields) merge into the wait-for graph; no cycle -> read the predicate ledger | The cycle is a relation, not a point; the ledger is history’s only admissible witness |
| 4 Close | Fix + mechanical regression (the TSan report goes from present to absent) | Passing a rerun isn’t regression; it’s luck |
On the prevention side, the point is this: each of the five diseases has its own vaccine, and one vaccine does not cover five diseases.
- Deadlock: a lock-hierarchy discipline (one level per lock, acquisitions only escalate, statically checkable) or
std::scoped_lockfor one-shot multi-lock acquisition (C++17, deadlock-avoidance algorithm inside); the Windows equivalent is a fixedEnterCriticalSectionsequence plus review discipline. A TSan concurrency build in CI reportslock-order-inversionbefore merge. - Lost wakeup: the
while (!pred) waitiron rule, no exceptions; the predicate must be protected by the same mutex as the condition variable. - Livelock: backoff plus a retry cap on retry paths — the scheme Ethernet has used for fifty years. Randomize the backoff on both sides and the phase difference disperses on its own.
- Starvation: use queued or fair locks where fairness matters — or at minimum expose per-thread acquisition counts as a metric. Starvation is diagnosed from distributional data, so make sure the distribution exists.
- Phantom hang: observability work — split the progress heartbeat by dimension (business progress, I/O progress, GC share) so the “zero progress” reading is naturally multidimensional. A service with a single heartbeat line is a service that has saved the phantom hang for itself.
- Organization: put the three-layer triage on page one of the on-call handbook (one
pscommand is worth half an hour of guessing); hook ProcDump’s hang trigger (unresponsive window) to automatic collection and feed the dumps into the analysis pipeline.
Finally, the one discipline that applies uniformly to all five: settle which dimension is motionless, for which participant, and in which cell of the timeline the cause sits — before you reach for a tool.
11. Coda: Crashes Lie in Space, Hangs Lie in Time
Look back at the premises this series has dismantled. Bugs Written by AI Look Like Correct Code dismantled “the author knows the intent.” The Harder You Look, the Less It’s There dismantled “observation doesn’t change the system.” This piece dismantles the most everyday one of all: “the program stopped, so the program is broken.” Zero progress can mean a cycle, a miss, a spin, an injustice — or simply that your yardstick was measuring the wrong dimension.
Put crash and hang side by side and you get a couplet: crashes lie in space (the line that crashes is not the line that was wounded); hangs lie in time (standing still in the present is not having your cause in the present). Their confluence is the completed form of what the heisenbug piece kept saying: debugging is a conversation with a system that changes because of you, and what you read is always a projection cast by your observation — a crash’s projection is displaced along the axis of space, a hang’s along the axis of time.
Tools will keep changing: the futex owner field, !cs -o‘s owner stacks, TSan’s lock-order graph — in ten years these may all go by different names. But this differential diagnosis table will not age, because it is built not on any tool but on one plain fact: “motionless” is a measurement you took, not a state the system is in. The next time you face a “hung” process, say that sentence to yourself first, and then ask three questions — motionless on which dimension? whose cause? at which coordinate on the time axis? Answer those before you touch a tool.
Appendix A: The Hang Differential-Diagnosis Quick Reference
| Type | Tense of the cause | Nature of the scene | Triage fingerprint | First-choice forensics | Misdiagnosis trap | Immunity |
|---|---|---|---|---|---|---|
| Deadlock | Cancelled future | Frozen forever | CPU near 0, all threads S + futex waiting | Full-thread dump + wait-for graph + owner fields | Treating queued victims as culprits; invisible locks with no lock calls | Lock hierarchy / scoped_lock / TSan |
| Lost wakeup | A spent past | Already dead | Looks identical to deadlock; wait-for graph has no cycle | Read the predicate ledger; record-and-replay (TTD/rr) | Grabbing dumps over and over (information content: zero) | while + predicate |
| Livelock | Spinning present | Flowing | CPU pegged, R state | Sampling profiles + retry/success counter ratio | Reading high CPU as “busy doing real work” | Backoff + retry cap |
| Starvation | Perpetually deferred present | A distribution | Any single snapshot looks normal | Long-window counters, wait histograms, tail latency | Normal averages hiding one starving thread | Fair locks / starvation metrics |
| Phantom hang | In the observer’s expectations | On another dimension | CPU/IO/GC active, heartbeat alive | Change yardsticks (CPU/disk/network/GC/paging) | Grabbing a dump (no pathology in the scene) | Multidimensional progress metrics |
How to use it: run section 2’s three-layer triage first (one ps line), match your readings against the “triage fingerprint” column to settle the family, then act on the “first-choice forensics” column. When triage is inconclusive, start by assuming you are holding the wrong yardstick (row five) — it is the cheapest hypothesis you own.
Appendix B: Wait-Graph Evidence Commands
Linux (captured on Ubuntu 24.04):
top -H -p <pid> # triage layer 1: per-thread CPU (livelock pegged / deadlock zero)
ps -eLo pid,tid,stat,wchan:24,comm # triage layer 3: state letter + kernel wait channel
cat /proc/<pid>/status | grep -E "Threads|switch" # thread count + voluntary/involuntary switch counts
gdb -p <pid> -batch -ex "thread apply all bt" # all stacks: waiting posture + waiting target
gdb -p <pid> -batch -ex "p m1" # mutex evidence: __lock / __owner (the holder's TID)
gdb -p <pid> -batch -ex "p cv" # condition-variable internals (the faint trace of history)
pstack <pid> / eu-stack -p <pid> # lightweight all-stack capture (when gdb feels too heavy)
perf record -F 997 -p <pid> + perf report # sampling profile: the livelock's flowing scene
setarch $(uname -m) -R ./prog # disable ASLR before running TSan on WSL2
Windows (captured on Win11 + cdb):
Task Manager -> Details -> Add Columns (CPU, Threads, I/O) # triage layer one
procdump -ma <pid> hang.dmp # full-thread dump (when the scene is frozen)
procdump -h <exe> hang.dmp # auto-capture on unresponsive window (phantom-hang probe)
cdb -z hang.dmp -c "~*k; !cs -l; q" # all stacks + list of locked critical sections
!cs <addr> # one critical section: LOCKED / OwningThread / LockCount
!cs -o <addr> # the trump card: prints the owner's stack directly
!cs ntdll!LdrpLoaderLock # autopsy the loader lock (it is the struct body in
# modern ntdll -- no poi)
dt ntdll!_RTL_CRITICAL_SECTION <addr> # read the fields by hand (needs symbol-server ntdll)
Notes: the !cs family needs type-complete ntdll symbols from a symbol server; export-only symbols produce “Bad symbols for NTDLL.” On Windows 11, !cs -l surfaces only the few critical sections that still carry DebugInfo — for a full inventory of your locks, verify each one with !cs -o <your lock's address>.
Appendix C: A Field Guide to Invisible Locks (for When the Cycle Contains None of Your Locks)
| Invisible lock | Who holds it | The classic collision |
|---|---|---|
| loader lock (ntdll!LdrpLoaderLock) | the system loader (for the duration of DllMain / LoadLibrary) | CreateThread + Wait inside DllMain; calling any API that may load a DLL from DllMain (COM, registry, some CRT functions) |
| process heap lock (allocator internals) | malloc / HeapAlloc itself | one thread preempted inside malloc while another mallocs from a callback that holds your lock |
| CRT initialization lock | C runtime startup code | static-object constructors interleaving with DllMain / thread startup |
| environment-variable lock | getenv / putenv | a thread holding it calls your callback, which takes your lock (a classic lock-inversion source) |
| GDI / USER32 internal locks | the windowing subsystem | window procedures and message pumps across threads, or holding a GUI lock into business code |
| your dependency’s internal locks | third-party libraries | the library calls your code while holding its lock, your code takes yours — the cycle closes across the library boundary |
The diagnostic essential: invisible locks appear nowhere in your code — only on somebody else’s stack. When hunting the cycle, don’t stare only at explicit signals like EnterCriticalSection or pthread_mutex_lock; look at which lock-holding paths each stack passes through (loader-initialization frames, allocator-internal frames, CRT startup frames), and recover the hold edges via !cs / __owner. Whether a stack holds a lock is a question of semantics, not of function names.
Related Articles
- The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging – how observation changes the system: the epistemological root of triage and “change yardsticks”
- Non-Invasive Probes: How ASan/TSan/UBSan Expose Bugs Before They Ever Misbehave – the full theory behind TSan’s lock-order-graph verdict
- Bugs Written by AI Look Like Correct Code: An Epistemic Rewrite of Debugging – where the series’ epistemology starts
- Visual Studio Debug vs Release, Decoded (Chinese) – the environmental background of the timing family
- WinDbg for User-Mode Debugging: A Field Guide (Chinese) – a systematic tutorial for Appendix B’s commands
Leave a Reply