Every detective show teaches the same lesson early: the place where the body turns up is rarely where the murder happened. Heap bugs work exactly the same way. The line of code that finally crashes is usually just where the body was found. The actual crime scene — the instruction that wrote where it shouldn’t have — may sit hundreds of instructions upstream, on another thread, or days in the past.
This article is the forensics handbook for that kind of case: how to work backwards from a crash to the moment the memory was corrupted. It picks up where The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging left off — that piece explained why live observation is doomed; this one explains what to do with the evidence once you have it. (For how to collect the evidence itself, see “Dump Files: A Complete Field Guide” — not yet published; link to follow.) Every Windows-side output in this article was captured live on Windows 11 + MSVC 2026, and every Linux-side output on Ubuntu 24.04 + glibc 2.39; all programs are reproducible.
Start with three incident reports. Same underlying disease — memory overwritten by an out-of-bounds or dangling write — three completely different deaths:
Report 1: dies at the hands of a sanity check
HEAP CORRUPTION DETECTED: after Normal block (#92) at 0x0000013AC5BB4B60.
CRT detected that the application wrote to memory after end of heap buffer.
Report 2: no crash at all, but the books are quietly wrong
victim@000001AFEB6D4E60 stale@000001AFEB6D4E60
victim->hits = 0x7AB7AB7A <- killer's fingerprint
Report 3: dies inside a completely innocent line of code
strlen (0x7ffb2c1a4d20) # top of stack, a libc string function
process_data (main.cpp:141) # your code -- but this line only touched the body
In the first, the error fires on the free call — while the code that did the damage is miles away. In the second, the program runs happily except some field keeps showing up with a value nobody ever wrote. The third is the most deceptive: the top of the stack is a perfectly ordinary strlen, and the line of your code beneath it looks completely fine — because it is fine. It just happened to brush against the wound.
Three deaths, one killer. This article answers a single question: how do you find the perpetrator from scenes like these.
1. First, Fix a Reflex: The Stack Points at the Victim, Not the Killer
“Look at the crash stack” is lesson one of debugging, and the habit is fine. What’s broken is the assumption underneath it: where it crashes is where the bug is. For bugs that die on the spot — a null dereference, say — the assumption holds. For heap corruption, it fails systematically.
Here’s why. A heap-corruption incident has three roles, in three different places:
| Role | What it is | Location |
|---|---|---|
| Perpetrator | The instruction that actually wrote bad memory | Far upstream in time |
| Victim | The object whose memory got trashed | The crime scene |
| Crash site | Where the program finally hits the wound | Far downstream in time |
Beginners treat the crash site as the crime scene. So they crash inside strlen and suspect their string-handling code. They crash inside free and suspect their memory management. They crash inside malloc (glibc users know malloc(): corrupted top size by heart) and glare at the allocator. All wrong. Most of the code on the crash stack is the victim, not the killer. At the moment the killer struck, nothing crashed, nothing complained — the program kept running quietly while the perpetrator walked away.
How do you tell a stack worth investigating directly from a stack that’s just a victim filing a report? Two checks:
- Is the top of the stack your code, and is that line writing to memory? If the crash happens inside your own assignment, memcpy, or container push — that’s likely the actual crime scene; investigate. If it happens inside a libc string function, a container lookup, or allocator internals — that’s a victim reporting, and the killer isn’t in the building.
- The usual-suspects list. A whole family of functions is almost always innocent:
strlen,strcmp,memcpy, map/unordered_map lookups, string operations, and the consistency checks insidemalloc/free. Their shared trait: they only read, or they only touch memory exactly as contracted. When they crash, the memory handed to them was already damaged.
This is the flip side of the heisenbug problem from my earlier piece: the heisenbug’s difficulty is “observation changes the system”; heap corruption’s difficulty is “the causal chain is severed in time.” The crash stack can only ever show you T3 — the moment of the report. What you’re hunting is T0 — the moment of the crime.
2. The Silent Interval: Why Heap Corruption Always Surfaces Late
Lay the life of a heap corruption onto a timeline:
T0 T1 T2 T3
the crime the body is reused a victim touches crash / error
(bad write) (wound overwritten the wound (you finally find
or relocated) out anything)
|<------------- silence: the system runs normally, wounded ---------->|
How far apart can T0 and T3 be? Arbitrarily far. Hundreds of thousands of instructions, hundreds of call frames, countless thread switches, days of uptime. That’s not rhetoric — it follows directly from how the heap works. Let’s prove it with an experiment.
Experiment: The Crime Happens in a Loop, the Report Fires on free
The full program (MSVC debug CRT), copy-paste reproducible:
// lag.cpp - the silent interval: crime first, report much later
#include <cstdio>
#include <cstring>
#include <crtdbg.h>
int main() {
// Send CRT reports to stderr (keeps dialogs out of the way)
_CrtSetReportMode(_CRT_WARN, _CRTDBG_MODE_FILE);
_CrtSetReportFile(_CRT_WARN, _CRTDBG_FILE_STDERR);
_CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_FILE);
_CrtSetReportFile(_CRT_ERROR, _CRTDBG_FILE_STDERR);
_CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
_CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDERR);
char* a = new char[32]; // the future perpetrator overflows here
char* b = new char[32]; // innocent neighbor
char* c = new char[32]; // another innocent neighbor
strcpy_s(b, 32, "neighbor-B");
strcpy_s(c, 32, "neighbor-C");
// ---- T0, the crime: loop bound typo (<=), one byte past the block
for (int i = 0; i <= 32; ++i) a[i] = 'X';
// ---- silence: the damage exists, everything looks fine
printf("[silence] a overflow done, neighbors intact: b=[%s] c=[%s]\n", b, c);
for (int k = 0; k < 5; ++k) {
char* t = new char[32];
strcpy_s(t, 32, "busy-but-fine");
printf("[silence] alloc #%d ok\n", k);
delete[] t;
}
// ---- T3, the report: CRT checks the no-man's land only on free
printf("[report ] about to delete a ...\n");
delete[] a; // <-- HEAP CORRUPTION DETECTED fires here
printf("[report ] never reached\n");
delete[] b;
delete[] c;
return 0;
}
Build and run (command line: cl /utf-8 /Od /EHsc /MDd /RTC1 lag.cpp, or any Debug-configuration project):
[silence] a overflow done, neighbors intact: b=[neighbor-B] c=[neighbor-C]
[silence] alloc #0 ok
[silence] alloc #1 ok
[silence] alloc #2 ok
[silence] alloc #3 ok
[silence] alloc #4 ok
[report ] about to delete a ...
HEAP CORRUPTION DETECTED: after Normal block (#92) at 0x0000013AC5BB4B60.
CRT detected that the application wrote to memory after end of heap buffer.
[report ] never reached
Walk the tape frame by frame:
- T0 is the
forloop. Thei <= 32typo writes byte 33 (index 32) one byte past the block — landing exactly on the guard fence the debug CRT pads around every allocation. - Six lines of silence, all green. Neighbors b and c are intact; five allocate/free cycles succeed. The wound exists; nobody has touched it.
- T3 is
delete[] a. The debug CRT validates the fence at free time, finds it trashed, and reports. Note the distance: the crime and the report are separated by ten lines of code and six heap operations — and I deliberately compressed it. In a real project, T0 and T3 routinely sit in different modules, written by different people, thousands of lines apart.
(Aside: this CRT build prints the report and then keeps executing — which is why the frees of b and c still run. Some CRT versions abort outright. Either way, the line that reports is T3, never T0.)
Three Delay Mechanisms
Why does heap corruption inevitably surface late? Because the heap’s structure guarantees the wound precedes the symptom:
Mechanism 1: whoever writes the damage doesn’t read it; whoever reads it didn’t write it. The perpetrator’s code writes and moves on. The injured object waits for the next passer-by — a log formatter, a container iteration, a callback. Perpetrator and victim can sit an entire call graph apart.
Mechanism 2: the heap is a crime scene that cleans itself. This is the most overlooked and the most lethal property. malloc’s reuse replaces the body outright — the same address gets handed to someone else, and every trace of the wound is overwritten with fresh data. free’s coalescing glues adjacent free chunks together, changing the wound’s shape. Time is not a neutral bystander: time destroys evidence for the killer. This is why you take a dump immediately (see “Dump Files: A Complete Field Guide”, not yet published), and why “let it keep running and see what happens” is a terrible instinct.
Mechanism 3: structural wounds only detonate when the allocator itself opens the books. The allocator keeps ledgers on free blocks — sizes, neighbor pointers, status bits. Trash an allocated object’s data and nobody cares for now; trash the ledger, and nothing happens until the next malloc/free walks that page of the books. The crash then lands inside libc or the runtime, on a stack of code you don’t recognize. glibc users know this family of errors well: malloc(): corrupted top size, free(): invalid next size, and friends. Their shared property: the report location has nothing to do with the crime location — only with when the ledger got audited.
3. Marks on the Body: The Fill-Pattern Fingerprint Catalog
Around 2008, the MSVC CRT team did something genuinely kind: in debug builds, the runtime fills heap memory with a fixed pattern for every lifecycle state. They probably didn’t realize that a mechanism designed to catch “use of uninitialized memory” doubles as a complete set of autopsy tags for the rest of us.
In the heisenbug article I treated these fills as “well-meaning lies that mask bugs” — reading 0xDDDDDDDD in debug happens not to crash, while release reads random garbage and detonates. That’s the observer-effect lens. Now switch lenses: the same mechanism is also evidence. When you see these values in memory, you don’t guess — you know what that memory was in its past life.
Experiment: Identifying the Body, Three Ways
// fill.cpp - fingerprint experiment: fill patterns (MSVC debug CRT)
#include <cstdio>
int main() {
// 1) freshly allocated, never written heap memory
unsigned int* p1 = new unsigned int[4];
printf("fresh alloc : %08X %08X %08X %08X\n", p1[0], p1[1], p1[2], p1[3]);
// 2) look again after the free (reading freed memory to see the tags)
delete[] p1;
printf("after delete: %08X %08X %08X %08X\n", p1[0], p1[1], p1[2], p1[3]);
// 3) uninitialized stack memory
unsigned int s[4];
printf("stack uninit: %08X %08X %08X %08X\n", s[0], s[1], s[2], s[3]);
return 0;
}
Build and run (note: the stack fill needs /RTC1, which the IDE’s default Debug configuration includes but the command line doesn’t):
fresh alloc : CDCDCDCD CDCDCDCD CDCDCDCD CDCDCDCD
after delete: DDDDDDDD DDDDDDDD DDDDDDDD DDDDDDDD
stack uninit: CCCCCCCC CCCCCCCC CCCCCCCC CCCCCCCC
Three values, three one-liners:
| Value | State | Mnemonic |
|---|---|---|
0xCD | Heap, allocated, never written | Clean: freshly cleaned room |
0xDD | Heap, freed | Dead: the body |
0xCC | Stack, uninitialized locals | Clean stack; also, literally, the x86 int3 breakpoint instruction — which is why trashed stacks sometimes “randomly break into the debugger” |
The practical value is hard to overstate. Find 0xDDDDDDDD where a pointer should be and you know, without reading a single line of code, that someone has a use-after-free. Spot 0xCDCDCDCD near a crash and the verdict writes itself: uninitialized heap memory being read. You can even read the order of events — which is exactly the next section’s business.
The NT heap (the operating system’s own HeapAlloc family) has its own tag set: 0xABABABAB (allocated, uninitialized), 0xBAADF00D (Bad Food, LocalAlloc uninitialized), 0xFEEEFEEE (after HeapFree). glibc doesn’t fill by default, but one environment variable fixes that: MALLOC_PERTURB_=165. Live-tested on glibc 2.39, with a boundary worth memorizing: large blocks (above 0x408 bytes) get 0xA5 filled across the body on free and its bitwise complement 0x5A on the next allocation — a body carrying two distinct postmarks, “freed” and “reborn.” Small blocks take the per-thread tcache path with no filling at all, and their first 16 bytes are occupied by list pointers (a dangling read that sees garbage instead of A5 is very often exactly this). The full cross-platform table is Appendix A; keep a copy and match against it at the scene.
The Limits of Fingerprints, and Voluntary Tattooing
The fill catalog has one innate weakness: it only exists in debug builds. Release CRT doesn’t fill, NT heap mostly doesn’t, glibc won’t unless told. And as Debug vs Release, Decoded (Chinese) explains, heap bugs love hiding in release. Hence the second half of this section’s title — voluntary tattooing:
- MSVC: keep a Debug test suite on CI specifically to cash in on the fills (build-matrix strategy in section 8 of Non-Invasive Probes).
- glibc:
MALLOC_PERTURB_=165. One environment variable, release-grade fingerprints. - The endgame: ASan — allocations default-filled with
0xBE, freed blocks quarantined and poisoned so dangling accesses can’t miss. Its machinery is covered in depth in Non-Invasive Probes: How ASan/TSan/UBSan Expose Bugs Before They Ever Misbehave; the one thing to note here is that ASan is essentially the official upgrade of “tattoo every body, and forbid destroying them.”
4. Wound Morphology: Inferring the Weapon from the Shape of the Damage
The autopsy report is in; now for the forensic examiner’s core skill: reading the weapon from the wound. Different ways of corrupting memory leave differently shaped scars. This is the densest section of the article — six wound shapes, six suspect classes.
First, the minimum viable heap knowledge (forensics only; no allocator internals beyond what you’d need at a scene):
- glibc: every chunk carries a 16-byte ledger up front —
prev_size(8 bytes) thensize(8 bytes, low 3 bits are flags). The pointer malloc hands you is ledger-plus-16. Sop[-16..0)is your own chunk’s ledger, andp[usable_size..)is the next chunk’s ledger. The first thing an overflow hits is the books, not the neighbor’s data. - Windows NT heap: same shape — header first, data after; the header records size and state, and the allocator only notices when it does its bookkeeping.
- tcache (glibc 2.26+): per-thread cache of small chunks. Freed chunks hang on a linked list whose pointers live at the start of the chunk — i.e., exactly where your “first field” is. That detail pays off shortly.
Shape 1: One neighboring field, or a few bytes, damaged — suspect: off-by-one
The classic wound: for (i = 0; i <= n; ++i), a memcpy length off by one, buf[n] bound typos. Tiny scar, tight against the end of the block, often only touching the next chunk’s prev_size or the low byte of its size. There’s a nasty glibc wrinkle: if that byte happens to shrink the size, free “succeeds” and the chunk goes back in the cache as a smaller block — the books are now wrong, but nobody complains. The heap’s notion of layout quietly diverges from reality, sometimes ending in overlapping chunks and a completely unrelated crash much later. The security world calls this the poison null byte and builds exploitation chains on it; for engineers the lesson is that a shape-1 wound can lie dormant until it has completely changed shape.
Shape 2: A large region overwritten with one repeating value — suspect: the memset/strcpy family
The wound is a sea of identical bytes — say, a flood of 0x41 (hex of ‘A’). This is a “write-through” injury: source data flattens everything after it until it hits an unmapped page. Paradoxically, the easiest case to work — the wound is huge, one !heap -srch (WinDbg) or gdb find pins the blast radius, and the writing code usually has visible memcpy/memset fingerprints nearby. Rule of thumb: the value filling the wound is a sample of the killer’s handwriting (it came from the killer’s source data).
Shape 3: Ledger fields (size, next pointers) surgically altered — suspect: an adjacent chunk’s overflow
Wound signature: the data area is pristine, but the size field holds an absurd value, or a free-list / tcache pointer aims somewhere that doesn’t exist. These wounds are always committed by the neighbor — note well, not by the owner of the damaged block. All reports come from the allocator’s ledger checks; see Appendix B for the glibc error fingerprint table. Here’s a glibc reproduction (gcc -O0 topbreak.c, glibc 2.29+):
// topbreak.c - the crime is in memset, the report is in malloc
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void) {
char* a = malloc(0x18);
// a's chunk is 0x20 bytes; a+0x18 is the *next* chunk's size field
// (the next chunk, right now, is the top chunk)
memset(a + 0x18, 0x41, 8); // T0: off-by-8 trashes top's size
// status report goes to stderr: printf's first output needs to malloc
// an stdio buffer, and the ledger is already broken -- the messenger
// dies on the spot inside malloc
fprintf(stderr, "crime committed, heap silent\n"); // silence
char* d = malloc(0x100); // T3: dies inside libc's malloc
printf("never reached\n");
return 0;
}
crime committed, heap silent
malloc(): corrupted top size
Aborted (core dumped)
The crime is in memset; the report is two statements later, in malloc, with the top of the stack buried in libc. Scale “two statements” to a real codebase and it becomes “two days.”
The output also hides a trap I stepped into live and is worth its own paragraph: if you send the silence line through printf to stdout instead, it mostly never appears — printf‘s first output has to malloc an stdio buffer, the ledger is already broken, and the messenger dies inside malloc before delivering a single word. After the crime, even stdout can’t be trusted.
Shape 4: Reading structurally-valid-but-nonsensical objects — suspect: use-after-free reads
Symptoms: an object’s fields make no sense — version numbers in the billions, pointers into the stratosphere, strings that look like line noise — but nothing crashes; the data is just wrong. Mechanism: the memory was freed and handed to someone else, and you’re reading the new tenant’s data, interpreted through the old tenant’s format. The mildest and most expensive shape: no crash, no trace, silent data corruption that surfaces in downstream reconciliation. In debug builds this shape gets betrayed by the fill patterns — you read 0xDDDDDDDD (reading a dead body) or 0xCDCDCDCD (reading a new tenant’s unwritten memory).
Shape 5: Your old fields show up in someone else’s new object — suspect: use-after-free writes
The one shape where the killer leaves a fingerprint on purpose — and the one most worth memorizing. Watch it happen, captured live:
// reuse.cpp - fingerprint transfer: on a release heap, the killer's value
// shows up inside an innocent new object. Build: cl /O2 /MD reuse.cpp
#include <cstdio>
#include <cstring>
#include <cstdlib>
struct Config {
long hits;
char name[24];
};
int main() {
Config* cfg = (Config*)malloc(sizeof(Config));
cfg->hits = 42;
strcpy(cfg->name, "prod-config");
Config* stale = cfg; // a cached old pointer, somewhere
free(cfg); // NT heap free: contents stay, untouched
stale->hits = 0x7AB7AB7A; // T0, the crime: a dangling write into freed memory
// Same-size allocation: the freed block is very likely handed right back
Config* victim = (Config*)malloc(sizeof(Config));
printf("victim@%p stale@%p\n", (void*)victim, (void*)stale);
printf("victim->hits = 0x%08llX", (unsigned long long)victim->hits);
return 0;
}
victim@000001AFEB6D4E60 stale@000001AFEB6D4E60
victim->hits = 0x7AB7AB7A <- killer's fingerprint
Look at the two addresses: identical. victim is a brand-new, innocent object. Nobody ever assigned to its hits field. Yet it was born with hits == 0x7AB7AB7A — the killer’s handwriting (stale->hits = 0x7AB7AB7A), left on the body before the victim ever moved in. A release heap doesn’t scrub freed memory and prefers to reuse same-size blocks, so the killer’s mark transfers, intact, to the next tenant.
The investigative value is enormous: the value inside the wound points straight at the killer’s code. In production, when a field shows up holding “a value nobody wrote,” skip the superstition — grep the codebase for the value (or for whatever produces it), and odds are you’ll land on the dangling write. The debug build stages the same scene even more beautifully, with two kinds of handwriting side by side on the body:
corpse : 7AB7AB7A DDDDDDDD DDDDDDDD DDDDDDDD
The killer’s fresh 0x7AB7AB7A right next to the deceased’s 0xDDDDDDDD tag — the sequence of the crime reads itself: death first (DD), the killer’s stroke second (7A).
Shape 6: Free-list / tcache pointer damage — suspect: double free or freeing a wild pointer
Wound signature: the error text mentions tcache, fastbin, or unsorted — allocator internals — or malloc returns an address that has no business existing. Mechanism: a double free hangs the same chunk on the idle list twice, or an overflow trashes a list’s next pointer; the allocator then follows a poisoned chain and hands out a block the books call free but reality calls occupied. glibc 2.26+ catches the simplest case on the spot:
char* p = malloc(0x20);
free(p);
free(p); // -> free(): double free detected in tcache 2
The only shape that routinely reports at the moment of the crime — tcache’s key check fires on the second free, at distance zero. But note the fine print: it catches immediate double frees. A double free with other allocations in between still takes the shape-3 route and explodes much later.
The Suspect Lineup, Condensed
| Wound shape | Suspect | First verification move |
|---|---|---|
| 1..n adjacent bytes trashed, usually at block end | Off-by-one / small overflow | Red-zone tooling (ASan / PageHeap) |
| Large region of one repeating value | memset/strcpy family | The value itself is the fingerprint — grep for it |
| Data fine, ledger trashed | Neighbor overflow | Data breakpoint guarding the size field |
| Weird values, no crash | UAF read | Debug fills / MSan |
| Old fields in new objects | UAF write | Grep the field’s value to locate the killer |
| Error text mentions tcache/fastbin | Double free / wild free | Audit the free paths |
5. Case Study: A Body That Turned Up Inside malloc (End to End)
Assemble the pieces into a full investigation. The case is the production-grade version of the last section’s experiments: a hot-reload config scenario where the old config is freed while still being written.
Background (the experiment code with the probes stripped to the skeleton):
struct Config {
long hits; // a counter: bumped once per request
char name[24];
};
Config* g_cfg;
void worker_step() {
g_cfg->hits++; // hot path: runs on every request
}
void reload_config() {
Config* fresh = load_new(); // read the new config file
delete g_cfg; // free the old config
g_cfg = fresh; // swap
}
One day, monitoring starts reporting “impossible” data: brand-new objects whose hits field is born with the value 0x7AB7AB7A (shape 5’s fingerprint). Sporadic, unreproducible, three or four times a month. That’s our cold case.
The wrong investigation (the path nine out of ten people take): set a breakpoint where the bad data is read, rerun, wait for a repro. You’ll wait forever — the breakpoint is guarding the body, not the killer; and every run reshuffles the heap layout, so which object receives the wound is a dice roll. (This is the everyday form of the “memory layout” layer from “Works on My Machine: A Taxonomy of Environment Differences”, not yet published.)
The right investigation, four moves:
Move one: examine the wound (don’t chase anyone). The bad value 0x7AB7AB7A is stable and always lands at offset zero of the object — shape 5, a UAF write; the killer is a stale reference still writing to the old object. Search the codebase for the constant: no literal match for 0x7AB7AB7A, but there is exactly one writer of hits in the entire codebase: g_cfg->hits++. Killer profile complete: a stale reference that didn’t follow the g_cfg swap. Now enumerate the places g_cfg gets cached (local copies, lambda captures, callback closures, another thread mid-read) — the suspect pool shrinks from the whole repo to a handful.
Move two: stake out (guard the wound, not the body). In a test environment, set a data breakpoint on the address of hits — a write watchpoint, guarding “who writes this memory,” not “who reads it.” Visual Studio: New Data Breakpoint; WinDbg: ba w8 <addr>; gdb: watch -l *(long long*)addr. It fires on the write itself; the top of the stack is the killer’s instruction.
Move three: if the stakeout never triggers (low frequency, needs load), bring in the page heap. For overflow-type killers, full PageHeap is a decapitation strike — see the live comparison in section 7. For UAF writes (an illegal write to a legal address), gflags /p /enable app.exe /full works too: each block gets its own page, freed pages are reclaimed, and the dangling write hits an inaccessible page on the spot. Distance: zero.
Move four: close the case and debrief. The killer: during reload_config, a thread had loaded g_cfg into a register and kept incrementing the old object after the swap. The fix: shared_ptr with atomic swap — reference counting keeps the old object alive until its last user lets go. The timeline at closing: T0 in the millisecond of the hot reload, T3 days later in data reconciliation, a million requests in between. No ordinary technique lets you see T0 from the scene at T3 — which is exactly why the methodology has to change.
(The Linux equivalent: MALLOC_PERTURB_=165 for fingerprints, gdb‘s watch -l for the stakeout (live capture in section 7), ASan for the three-stack verdict — allocation, free, and access sites handed to you at once. Different tools, identical plot.)
6. The Method: From “Reading the Crash Site” to “Reconstructing the Causal Chain”
The four-step investigative method, one table, worth pinning to your monitor:
| Step | Police work | Debugging action | Key insight |
|---|---|---|---|
| 1 | Secure the scene | Dump first; don’t poke the live process | Time destroys evidence: every allocation washes the scene |
| 2 | Autopsy | Classify the crash site: victim or suspect? | Your own write on the stack? Investigate. Otherwise treat as an incident report |
| 3 | Identify the victim | Read fill fingerprints and wound shape; classify the corruption | The values on the body talk: DD/CD/repeating floods/old fields |
| 4 | Stake out the killer | Data breakpoints on the wound / PageHeap / ASan | Move the exposure point onto the crime point: distance zero |
Underneath the table sits the article’s unifying idea. Look back: every heap-debugging technique ever invented is doing the same thing — compressing the time distance between the moment of damage and the moment of exposure:
| Technique | What it does | Crime-to-exposure distance |
|---|---|---|
| Naked heap (release, no tools) | Run wounded; the wound gets washed away | Astronomical: days |
| Fill patterns (debug CRT / MALLOC_PERTURB_) | Tattoo the body so it’s recognizable | Shortened: at least you can autopsy |
| Guard zones (0xFD / _CrtCheckMemory) | A doorbell on the wound | Rings only when the fence is touched, checked at free |
| Data breakpoints (ba w / watch) | A camera on the wound | Whoever writes gets caught, if you can wait |
| Full PageHeap | Private page per block, guard at the tail | The overflow crashes on contact: distance ~0 |
| ASan red zones + quarantine | Damage reported instantly, body preserved | Distance = 0, evidence never lost |
| TTD / rr recording | Everything rewindable | Rewind from T3 back to T0 at will |
The three weapons from the heisenbug article fall into place: dumps are the evidence bag, sanitizers are the live broadcast, reverse debuggers are the time machine. And the forensics playbook in this article is the case procedure that decides which tool to bring, which evidence to read, and which address to stake out. They answer “how do we get evidence”; this answers “where to look once you have it.”
7. Why “Breakpoint at the Crash” Is Doomed (The Beginner Trap, Examined)
Worth its own section, because it’s the most common move in heap-corruption investigations and the most predictably futile one. Three reasons, each more fundamental than the last:
Reason one: the crash site is the victim. Section 1 covered it. You’re staking out the crime scene waiting for the killer to return. The killer left the moment they struck; you and the body just stare at each other.
Reason two: reruns reshuffle the deck. Every run lays out the heap differently — ASLR, allocation order, environment sizes all decide whose doorstep the wound lands on. You’re waiting for a crash that will never take this exact shape twice. Standard heisenbug territory; see The Harder You Look, the Less It’s There, section 4.
Reason three: a breakpoint guards a place in space; the killer is a moment in time. An ordinary breakpoint’s semantics are “stop when execution reaches this instruction” — it assumes the killer comes back to this address. But a heap killer is “some particular write to some particular address at some particular time.” It does not re-enact itself. The semantics you need are “stop when this memory gets written” — that’s a data breakpoint (watchpoint), implemented in hardware via the debug registers. x86/x64 has exactly four of them (DR0-DR3), so which address to guard is a craft decision: first choice is the wound itself (the size field, the first 4/8 bytes of the corrupted field), never the whole object.
And the stakeout doesn’t always net the killer on the first cast. Here’s a live capture (Ubuntu 24.04 + glibc 2.39 + gdb 15; the program frees a block, then hands the dangling pointer to another function to write, with a watchpoint on the memory):
// stakeout.c - stakeout target: those 8 bytes, dangling-written after free
#include <stdio.h>
#include <stdlib.h>
static void worker(long long* stale) {
*stale = 0x7AB7AB7A7AB7AB7A; // the killer, hiding in another function
}
int main(void) {
long long* p = malloc(sizeof(long long));
*p = 42;
long long* stale = p;
free(p);
worker(stale); // the crime
return 0;
}
(gdb) break stakeout.c:14 # stop before the crime, set the watch first
(gdb) run
(gdb) watch -l *(long long*)stale
(gdb) continue
Hardware watchpoint 2: -location *(long long*)stale
Old value = 42
New value = 22906492249
#0 tcache_put (tc_idx=0, chunk=0x555555559290) at ./malloc/malloc.c:3166
(gdb) continue # top frame inside glibc: the undertaker
# hanging list pointers -- let him pass
Hardware watchpoint 2: -location *(long long*)stale
Old value = 22906492249
New value = 8842724935898475386 # 0x7AB7AB7A7AB7AB7A
#0 worker (stale=0x5555555592a0) at stakeout.c:7
#1 0x00005555555551ce in main () at stakeout.c:14
Two hits, one decoy: the first stop is tcache_put — the allocator hanging list pointers onto the body, a perfectly legal write, and the stack tells you so (top frame buried in malloc.c). Let it pass; the second hit is the killer: the new value is exactly the dangling-write constant, and the top frame is worker, the function that wrote it. What you filter from stakeout logs is not noise but undertakers — standard homework for watchpoint stakeouts. The classic beginner crash: seeing a wall of malloc.c frames on the first hit, assuming the tool misfired, and walking away.
And finally, the page heap’s decapitation strike, live comparison (Windows 11 + MSVC 2026; the program does one thing: malloc(8), then memset writes 9 bytes):
===== normal heap =====
p = 000001FF50C53BC0
still alive: the 1-byte overflow went unnoticed
exit=0
===== full PageHeap (gflags /p /enable heapguard.exe /full) =====
p = 0000026A9BC69000 # note: page-aligned, block tail hugging the page edge
VERIFIER STOP 000000000000000F: pid 0x5840: corrupted suffix pattern
process terminated on the spot # the instruction writing byte 9 dies right there
Same overflowing line: on a normal heap it passes silently (that one byte lands on padding or the neighbor’s header, waiting for some future free to maybe notice). Under full page heap, the block sits at the end of its own private page with an inaccessible page glued to its tail — the instant byte 9 is written, the process dies, and the crash site moves from “arbitrarily far downstream” onto the crime-scene instruction itself. (On recent Windows the page heap is implemented through the Application Verifier layer and reports corrupted suffix pattern; the classic implementation triggers a guard-page access violation instead. Different plumbing, same verdict: distance zero.)
The price tag matters too: under full page heap every allocation owns at least a page (4KB), a hundredfold-plus memory blowup. It’s a per-process, investigation-time-only tool, and every enable gets a matching gflags /p /disable when you’re done. Which is why it earns the label “siege artillery” here rather than “daily patrol” — the patrol shift belongs to the CI’s ASan builds and unit tests, with the full build-matrix setup in section 8 of Non-Invasive Probes.
8. Team Playbook: Turning Forensics into SOP
One person knowing the craft isn’t enough; the team should walk the right path on any heap crash. Four things:
1. First-pass triage, automated. When a crash report comes in, classify the stack first: top frame inside libc/string functions/allocator internals (the usual-suspects list) means it goes straight down the forensics route, with reruns-at-the-crash-point forbidden; top frame in your own code doing a write is the only case for the ordinary “read the stack, read the code” path. The automation is nearly free (regex on the top module), and it deletes the single largest block of wasted investigation time.
2. The five-minute first pass (three commands). With a dump in hand:
– WinDbg: !analyze -v for exception and stack -> .exr -1 for context -> db <addr> L20 on the suspicious pointer, then match raw bytes against Appendix A (DD/CD/FEEE give instant verdicts).
– gdb: bt for the stack -> x/16gx <ptr> for the memory -> info proc mappings to see where the address lands (heap? stack? unmapped?).
– Fingerprint identified? Jump straight to section 4’s lineup and pick a verification move. No fingerprint? Now you consider the stakeout.
3. Layered tool placement. Fill fingerprints (debug CRT / MALLOC_PERTURB_) cost nothing and live in CI permanently; the ASan build runs in CI daily (details in “Sanitizers” section 8); PageHeap opens only for battle and always closes after, written into the on-call handbook. Their crime-to-exposure distances, respectively: “autopsy possible” / “live broadcast” / “shot on sight.” Pick per incident.
4. Prevention closes the loop. The highest form of forensics is fewer cases: smart pointers and containers instead of raw new/delete (evaporating the suspect pool for shapes 4, 5, and 6 in one stroke); boundary-checked buffer interfaces everywhere; CI’s ASan + fuzzing so most overflows surface before merge (see “Sanitizers” for the fuzz-build setup). Every suspect class eliminated is one less midnight page for whoever’s on call.
9. Closing: The Time Dimension of Debugging
Step back and look at the arc of this series. “Bugs Written by AI Look Like Correct Code” dismantled “the author knows the intent.” The heisenbug piece dismantled “observation doesn’t disturb the system.” This one dismantled the most everyday assumption of all: “the crash tells me where the bug is.”
Assembled, they form one epistemological picture: the crash you see is never the bug itself — it’s the projection left behind when the bug collided with your observation method, your build configuration, your memory layout. Heap corruption takes this to the extreme, turning debugging from a question of space (which line) into a question of time (which moment, which writer, and why is it only exploding now). The crash stack is the present tense; the cause lives in the past tense; and the heap, your only witness, keeps destroying evidence.
So the next time a crash lands, recite the whole article in one line: is this where the body was found, or where the murder happened? Then bag the evidence, read the fingerprints, study the wound, set the stakeout. Tools will change, platforms will change. The procedure won’t.
Appendix A: The Fill-Pattern Fingerprint Table (Cross-Allocator)
| Value | Allocator / context | Meaning | What seeing it tells you |
|---|---|---|---|
0xCDCDCDCD | MSVC debug CRT heap | Clean: allocated, never written | Someone is reading uninitialized heap memory |
0xDDDDDDDD | MSVC debug CRT heap | Dead: freed | Use-after-free (read or write) |
0xFDFDFDFD | MSVC debug CRT heap | Fence: guard region around blocks | Out-of-bounds access into no-man’s land |
0xCCCCCCCC | MSVC debug stack (/RTC1) | Uninitialized locals | Reading uninitialized stack; prime suspect in trashed-stack cases |
0xABABABAB | Windows NT heap | Allocated, uninitialized | Reading HeapAlloc’d memory before writing it |
0xBAADF00D | Windows NT heap | LocalAlloc(LMEM_FIXED), uninitialized | Same (Bad Food) |
0xFEEEFEEE | Windows NT heap | After HeapFree | Use-after-free (OS-heap path) |
MALLOC_PERTURB_ value | glibc (with the env var set) | Large blocks: n on free, ~n on alloc; small blocks take tcache with no fill (first 16 bytes are list pointers) | A postmarked body (large blocks only): distinguishes reads-after-death from reads-after-rebirth |
0xBEBEBEBE | ASan | Default malloc fill (malloc_fill_byte) | Reading uninitialized memory in an ASan build |
| Shadow values 0xfd / 0xfa etc. | ASan | Shadow-memory poison marks (not the data’s own value) | The report text names them; no human eye needed |
Usage notes: the CRT/RTC family exists only in debug builds; NT-heap tags appear only on certain paths; glibc needs MALLOC_PERTURB_ set explicitly. In a release environment, the absence of fingerprints is itself information — it means it’s time to tattoo voluntarily (PERTURB / PageHeap / ASan).
Appendix B: glibc malloc Error Fingerprint Table
| Error text (malloc_printerr) | Literal meaning | Common cause |
|---|---|---|
free(): invalid pointer | Pointer isn’t a heap address, or badly misaligned | Freed a stack/global pointer, or a wild pointer |
free(): invalid size | This chunk’s size is illegal (too small/unaligned) | Neighbor overflow trashed this chunk’s ledger |
free(): invalid next size (normal) | Next chunk’s size is out of range | Size enlarged, or adjacent header damaged |
free(): invalid next size (fast) | Same, fastbin path | Same |
double free or corruption (top) | The freed chunk is the top chunk | Double free after the chunk merged into top |
double free or corruption (out) | Next chunk address lies outside the arena | Size enlarged; ledger badly damaged |
double free or corruption (!prev) | Next chunk says this one is already free | Double free (non-tcache path) |
free(): double free detected in tcache 2 | tcache key hit | The same chunk freed twice back-to-back (glibc 2.26+) |
malloc(): corrupted top size | Top’s size exceeds system memory | Overflow trashed the top chunk (glibc 2.29+) |
malloc(): invalid next size (unsorted) | Bad next size while taking from unsorted | Adjacent free-chunk header damaged |
malloc(): unaligned tcache chunk detected | tcache next fails de-obfuscation alignment | Dangling write trashed the tcache list (glibc 2.32+) |
malloc(): unsorted double linked list corrupted | unsorted list chain broken | Overflow trashed fd/bk pointers |
How to read this table: every one of these errors is a T3, never a T0. They tell you which part of the ledger is wrong (the wound shape). Combine with section 4’s lineup to profile the perpetrator, then deploy data breakpoints or the page heap to catch them in the act. Exact wording varies slightly across glibc versions; the noted versions are when each check appeared.
Appendix C: Heap Forensics Command Reference
WinDbg:
!heap -s # heap summary: sizes and growth trends
!heap -p -a <addr> # (page heap on) which allocation owns this address,
# plus its allocation stack -- the killer-catcher
!heap -h 0 # full block listing for the main heap
!heap -srch 41414141 # search all heaps for a value: scope a uniform flood
ba w4 <addr> # data breakpoint: fire when this address is written
dt _HEAP_ENTRY <addr> # read a chunk's ledger header
gflags /p /enable app.exe /full # turn on full PageHeap (wartime only)
gflags /p /disable app.exe # mandatory cleanup when done
gdb:
x/16gx <addr> # memory as machine words (fingerprint reading)
x/32bx <addr> # memory as bytes (single-byte wounds)
watch -l *(long long*)<addr> # hardware watchpoint: stop on any write
bt / frame N # stack / switch frames
info proc mappings # which region an address falls in (heap/stack/unmapped)
set env MALLOC_PERTURB_ 165 # voluntary glibc tattooing
run # run and wait for the hit
Visual Studio: Debug windows -> Breakpoints -> New Data Breakpoint (address expression) — the GUI equivalent of ba w. Note the whole process has only four hardware slots.
Related Articles
- The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging – the other half of the same story: how observation changes the system
- Bugs Written by AI Look Like Correct Code: An Epistemic Rewrite of Debugging – where the series’ epistemology starts
- Visual Studio Debug vs Release, Decoded (Chinese) – the full answer to why debug fills exist only in debug
- WinDbg for User-Mode Debugging: A Field Guide (Chinese) – a systematic tutorial for Appendix C’s commands
- “Dump Files: Principles, Types, Analysis, and Linux Practice” (not yet published) – the evidence bag: collecting and analyzing dumps
- Non-Invasive Probes: How ASan/TSan/UBSan Expose Bugs Before They Ever Misbehave – the everyday way to reach distance zero
- “Reverse Debugging: Principles and Use Cases” (not yet published) – the time machine: rewinding from T3 back to T0
Leave a Reply