Let’s start with an experiment.
// race_counter.cpp - a program that "looks fine"
#include <cstdio>
#include <thread>
long counter = 0; // plain variable: no lock, no atomic
void worker() {
for (int i = 0; i < 100000; ++i)
counter++; // two threads read-modify-write the same address
}
int main() {
std::thread a(worker), b(worker);
a.join(); b.join();
std::printf("counter = %ld\n", counter); // expect 200000
}
g++ -O1 -g race_counter.cpp -pthread -o rc
./rc
Run it a few times: 200000, 200000, 198743, 200000… occasionally a little short, mostly perfect. No crash, no error message, nothing visibly wrong. Most people run this experiment and conclude: “basically fine–it drops a few counts; if you really care, add a lock.”
But by the letter of the C++ standard, this program entered undefined behavior (UB) the microsecond two threads first executed counter++ together–two threads accessing the same memory with no synchronization, at least one of them writing. That is a data race, and the standard’s entire verdict on it is one sentence: a program with a data race has no semantics. “On x86 it only drops a few counts” is a gift from the hardware, not a property of the program. Swap in an ARM core with a weaker memory model, or a compiler version that hoists the load out of the loop, and the same code can go from “always just a few counts short” to “systematically losing half the increments” to occasional crashes. Whether the program is sick and whether you have seen symptoms are two independent questions.
Now run the same program with one different flag:
g++ -O1 -g -fsanitize=thread race_counter.cpp -pthread -o rc_tsan
./rc_tsan
Output (abridged):
==================
WARNING: ThreadSanitizer: data race (pid=31234)
Write of size 8 at 0x55f3... by thread T2:
#0 worker() race_counter.cpp:7
Previous write of size 8 at 0x55f3... by thread T1:
#0 worker() race_counter.cpp:7
Location is global 'counter' of size 8 in race_counter.cpp:4
Thread T2 (running) created by main thread at:
#0 pthread_create
#1 main race_counter.cpp:13
==================
counter = 200000 still prints. The program is “perfectly healthy.” But TSan has already written the verdict: line 7, two threads, writing the same memory, with no synchronization whatsoever between them. It didn’t wait for the bug to misbehave. It proved the bug exists.
This article is about the machinery behind that trick: how do ASan, TSan, and UBSan–collectively, sanitizers–get evidence without alarming the system? What exactly do they smuggle into your program? And a more fundamental question almost nobody bothers to ask–what does “non-invasive” actually mean?
I. First, Correct a Popular Claim: Sanitizers Do Change Timing
In the Heisenbug piece I wrote: “breakpoints change timing; probes don’t.” As a first-order approximation, that’s serviceable. Strictly speaking, it’s wrong–and wrong in a valuable way.
Check the disturbance inventory of each sanitizer:
- ASan changes memory layout. Every heap allocation gets redzones around it; freed blocks aren’t reused immediately but go into quarantine. Allocation addresses, object spacing, the heap’s growth trajectory–all different from a normal build.
- TSan changes timing. A 5-15x slowdown completely reshuffles how threads interleave. If a bug depends on a particular interleaving window, the TSan build either hits that window more often or makes it vanish entirely.
- UBSan changes things too. The extra check instructions alter code size, inlining decisions, and instruction-cache behavior.
In other words, sanitizers work precisely by perturbing the system on a massive scale. Their effectiveness does not come from “not touching anything.”
So why don’t they backfire the way breakpoints do? Because “what gets disturbed” was never the point. The point is the criterion in the next section.
II. The Core Criterion: The Variable Your Verdict Relies On Must Not Be the Variable You Disturbed
Split any observation technique into two halves:
- The perturbation term: what it changes about the system under test.
- The verdict term: what it bases its “bug found” verdict on.
Run the usual suspects through this frame:
- Breakpoints. Perturbation: freezing threads (timing). Verdict: you witnessed the timing window while present. The verdict rests on exactly the variable being disturbed–the moment you show up, the window is gone. Backfire. This is the mechanism behind “the harder you debug a Heisenbug, the further it retreats.”
- printf. Perturbation: timing + the heap’s allocation sequence (layout). Verdict: witnessing a window or a landing spot. Same backfire.
- Guard pages (the old Electric Fence approach): place each allocation at the end of a page, right against an inaccessible one. Perturbation: layout upheaval, orders of magnitude slower. Verdict: the overflow happens to cross the page boundary into the guard page. Overflow by one byte, landing spot still inside the same page? Missed. The verdict depends on “physical landing spot,” and the landing spot is exactly what was disturbed. Detection by luck.
- ASan. Perturbation: layout (redzones + quarantine). Verdict: at the moment an access happens, check that address’s poison marker in shadow memory. It doesn’t ask “which neighbor did you physically hit”; it asks “is this address inside this object’s own territory.” Whether the layout changed never enters the verdict. Verdict ≠ perturbation. The disturbance is background noise; the detection stands.
- TSan. Perturbation: speed (timing). Verdict: whether a happens-before partial order exists logically between two accesses. How they physically interleaved, how slow the run got–none of it enters the verdict. Stands.
- Core dump. Perturbation: none (the system is dead). Verdict: the snapshot of the body. Trivially stands.
- rr/TTD recording. Minimal perturbation during recording (writing the trace); the verdict happens during replay, and by then the timing is frozen in the trace–your observation can no longer rewrite it. Stands.
Collected into one table:
| Technique | What it disturbs | What the verdict rests on | Verdict depends on the disturbed variable? |
|---|---|---|---|
| Breakpoint / single-step | Timing | Witnessing the timing window | Yes–the harder you look, the further it moves |
| printf | Timing, layout | Witnessing a window / landing spot | Yes–the more you watch, the more it skews |
| Guard page | Layout | Overflow crossing a page boundary | Yes–misses depend on luck |
| ASan | Layout | Shadow poison marker (logical boundary) | No |
| TSan | Speed / timing | happens-before partial order (logical relation) | No |
| UBSan | Instruction layout | Type-system / standard-clause assertions | No |
| Core dump | None | Snapshot | No (trivially) |
| rr/TTD | Writing the trace (recording only) | Replay of frozen timing | No (decoupled after recording) |
One sentence to serve as the foundation of this entire article:
“Non-invasive” does not mean “doesn’t disturb the system.” It means “the assertion your detection relies on does not depend on the variable you disturbed.”
One level deeper: breakpoints and guard pages ask physical questions (“what coincidence just physically happened”), while sanitizers ask logical questions (“did this access violate a constraint the program is supposed to uphold”). Physical coincidences don’t reproduce; logical assertions hold on every execution. That explains the opening experiment: TSan never needed to wait for the physical event of “consequences becoming visible,” because “two unordered conflicting accesses” is a logical fact present on every single run.
The next three sections take apart how the verdict terms of ASan, TSan, and UBSan are actually built. Once you understand the machinery, you’ll know what every line of a report means, when to trust it, and when to be suspicious.
III. ASan: Swapping “Whom Did I Hit” for “Did I Step Into Poison”
Shadow memory: a registry entry for every byte
ASan’s verdict term is built on shadow memory: every 8 bytes of application memory maps to 1 byte in the shadow region, recording the “registry status” of those 8 bytes. On x86-64 Linux:
shadow_addr = (app_addr >> 3) + 0x7fff8000
The possible values of a shadow byte:
| Shadow value | Meaning |
|---|---|
0x00 | all 8 bytes addressable |
0x01–0x07 | only the first k bytes addressable (object tail) |
0xf1/0xf2/0xf3 | stack redzone (left/mid/right) |
0xf9 | global variable redzone |
0xfa/0xfb | heap redzone (left/right) |
0xfd | freed (quarantine)–the use-after-free marker |
The compiler inserts a few instructions before every memory access (load/store), logically equivalent to:
shadow = (addr >> 3) + 0x7fff8000;
if (*shadow != 0) // hit poison or partial addressability
__asan_report_load8(addr); // slow path: unwind, report, abort
// ...the original access follows
Note the character of these three lines: pure user mode, no syscalls, no locks, no thread ever frozen. The slow path is entered only when poison is hit; otherwise it’s a few wasted instructions. This is the physical form of “check the table and move on”–the check itself is deterministic and constitutes no intervention in timing.
Redzones and quarantine: turning two bug classes from “luck” into “certainty”
Redzones: on every heap allocation, ASan pads a redzone (tens of bytes) before and after the object, all marked 0xfa/0xfb. Any overflow–even by a single byte–necessarily lands in poison, regardless of “which physical neighbor is there.” Stack frames and globals get redzones too (0xf1–0xf3/0xf9).
Quarantine: when free/delete fires, ASan doesn’t return the memory to the allocator. It marks it 0xfd and parks it in quarantine (256MB total by default, evicted LIFO). Any read or write through a dangling pointer therefore necessarily hits the “freed” poison marker instead of freshly reused data.
Chain the two together and you get the essential difference between ASan and guard pages: a guard page’s verdict is “the overflow happened to cross a page boundary”–a physical-landing-spot question. ASan’s verdict is “is this address inside the bounds this object declared”–a logical-boundary question. The former asks physics; the latter asks logic. This is Section II’s criterion cashed out for ASan: yes, redzones disturb the layout, but the verdict reads the shadow registry, not the physical neighbors.
Example 1: a heap off-by-one
// asan_offby1.cpp
#include <cstdio>
int main() {
int* scores = new int[10];
for (int i = 0; i <= 10; ++i) // <=: one too many
scores[i] = i;
std::printf("done\n");
delete[] scores;
}
g++ -O1 -g -fsanitize=address asan_offby1.cpp -o ao
./ao
=================================================================
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000048
WRITE of size 4 at 0x602000000048 thread T0
#0 main asan_offby1.cpp:5
#1 __libc_start_main
#2 _start
0x602000000048 is located 0 bytes to the right of 40-byte region [0x602000000020,0x602000000048)
allocated by thread T0 here:
#0 operator new[](unsigned long)
#1 main asan_offby1.cpp:3
SUMMARY: AddressSanitizer: heap-buffer-overflow asan_offby1.cpp:5 in main
Read the report line by line:
WRITE of size 4 ... #0 main asan_offby1.cpp:5: the offending access, line 5, a 4-byte write.0 bytes to the right of 40-byte region [0x...,0x602000000048): byte-level precision. 40 bytes =10 * sizeof(int), and you wrote exactly byte zero past its right edge–the off-by-one measured for you. Guard pages are hopeless against “overflowed one step, still inside the same page.”allocated by thread T0 here: ... asan_offby1.cpp:3: the memory’s birth certificate. Overflow-class errors get two stacks (access + allocation); the use-after-free below gets three (plus the free site).
In a normal build this program is symptomless: scores[10] most likely lands in allocator padding or an adjacent object and quietly corrupts a piece of data nobody looks at. In the ASan build it is caught on the very instruction that performs the write–not the consequence becoming visible, but the act itself violating the rules.
Example 2: stack overflow and use-after-scope
// asan_stack.cpp
#include <cstdio>
#include <cstring>
void greet(const char* who) {
char buf[16];
std::strcpy(buf, who); // unbounded copy into a 16-byte stack buffer
std::puts(buf);
}
int main() {
greet("0123456789ABCDEF"); // 16 chars + '\0' = 17 bytes
}
==12346==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc...
WRITE of size 17 at 0x7ffc... thread T0
#0 strcpy (interceptor)
#1 greet(char const*) asan_stack.cpp:5
#2 main asan_stack.cpp:10
Address 0x7ffc... is located in stack of thread T0 at offset 48 in frame
#0 greet(char const*) asan_stack.cpp:4
This frame has 1 object(s):
[32, 48) 'buf' <== Memory access at offset 48 overflows this variable
Three details worth pausing on:
WRITE of size 17:strcpywas taken over by an ASan interceptor, so the 16 characters plus the trailing\0–17 bytes–are judged in one shot instead of byte by byte. Common libc functions (the memcpy/strcpy/strlen family) all have interceptors; coarser checking granularity, more complete reports.[32, 48) 'buf' <== Memory access at offset 48 overflows this variable: the report names the variable.buf‘s territory in the stack frame is the 16 bytes of[32, 48); your write landed at offset 48–territory boundary, overflow starting point, variable name, all in one line. In a normal build this is a silent stack corruption; in a probe build it’s an indictment.- Stacks get redzones too (
0xf1–0xf3). Two switches you have to turn on explicitly:-fsanitize-address-use-after-scopecatches “used after leaving scope” (out-of-scope variables are marked0xf8); stack use-after-return needs the runtime optiondetect_stack_use_after_return=1, because the physical stack frame is still there after return–a “fake stack” mechanism has to patch the semantics back in.
Use-after-free: a three-stack verdict
For UAF, ASan hands you all three stacks at once: the dangling access site, the free site, the allocation site. The Heisenbug piece shows this in full in its Section V; no repetition here. What deserves emphasis is quarantine’s value: without it, freed memory would be immediately reused by the next new, a dangling read could return “legitimate new data,” and nothing would ever fire. With quarantine, the freed block stays poisoned as 0xfd, and a dangling read necessarily hits the marker. The price is memory (one source of ASan’s 2-3x memory footprint: 1/8 shadow + redzones + quarantine).
Overhead and usage notes
- Roughly 2x slowdown, 2-3x memory. Slowdown: the shadow-check instructions before every access. Memory: shadow + redzones + quarantine.
- Use
-O1:-O0shatters accesses into fragments (slower, noisier);-O2may optimize away the very accesses that should have been caught. - MSVC (VS 2019 16.9+):
cl /fsanitize=address ...–available on the Windows side. - ASan is mutually exclusive with TSan/MSan (their shadow-memory address maps conflict); one per process.
IV. TSan: The Definition of a Data Race Never Mentions “Crash”
First, get the definition straight
The C++ memory model defines it: two threads access the same memory location, at least one access is a write, and no happens-before partial order exists between the two accesses. Note two features of this definition. First, it never mentions “crash,” “corruption,” or any consequence at all. Second, it is a logical-relation judgment, not a physical-time judgment. What clock time the accesses happened at, how far apart they were–none of that is in the definition. The definition asks exactly one thing: is there a logical guarantee of ordering between them?
Where do happens-before edges come from? Synchronization primitives: thread create/join, mutex unlock->lock, atomics, semaphores. If T1 unlocks m and T2 then locks the same m, everything T1 did inside the lock “happens-before” everything T2 does after acquiring it.
This language wasn’t invented for concurrency–it is the partial order from Lamport’s 1978 paper, Time, Clocks, and the Ordering of Events in a Distributed System, the cornerstone of ordering in distributed systems. A shared-memory multithreaded program is, logically speaking, a distributed system: physical clocks can’t be trusted; the only thing you can trust is causality. That vista deserves its own article; one sentence here as a placeholder.
Vector clocks: turning “who has seen whom” into something computable
TSan maintains a vector clock per thread (length = number of threads; slot i records “how many of thread i’s events have I seen”), and a few access-history slots per 8 bytes of memory (recording the thread ID and logical time of the most recent accesses). Synchronization primitives merge clocks: unlock publishes your clock onto the lock; lock merges the lock’s clock into yours. “Is there a happens-before edge?” thereby becomes a vector comparison.
Walk through a minimal example (threads T1 and T2):
No synchronization:
T1 clock (1,0), writes x -> history slot records [T1@1]
T2 clock (0,1), writes x -> checks history slot: last was T1@1,
but my clock has "never seen" T1's event 1
-> the two accesses are unordered
-> data race, report
With a lock:
T1 clock (1,0), writes x, unlock(m) (publishes (1,0) onto m)
T2 lock(m) (merges in (1,0) -> T2 clock becomes (1,1)), writes x
-> checks history slot: last was T1@1,
my clock has seen T1@1
-> partial order exists -> legal
Notice that “nanoseconds,” “window,” and “interleaving”–the vocabulary of physical time–never appears in the decision procedure. That is TSan’s verdict term: a purely logical partial-order check. What its 5-15x slowdown violently disturbs is physical timing, and physical timing isn’t in the verdict formula. Section II’s criterion, cashed out again.
Example 3 (the opening experiment, revisited): the race that never crashes
In the opening race_counter.cpp, every run oscillates between “perfect result” and “a few counts short,” and it will never crash. TSan reports, on every run:
WARNING: ThreadSanitizer: data race
Write of size 8 ... by thread T2: #0 worker() race_counter.cpp:7
Previous write of size 8 ... by thread T1: #0 worker() race_counter.cpp:7
Location is global 'counter' ...
Thread T2 created by main thread at: ...
The report carries complete stacks for both conflicting accesses, the memory’s ownership, and both threads’ birthplaces. The key to reading it correctly: “Previous write by thread T1” does not mean T1 finished writing before T2 started in any physical sense. It means that under no partial order, the two accesses cannot see each other. At the hardware level, x86 aligned 8-byte writes are atomic, so the consequences are mild. At the language level this is UB, and the compiler is entitled to optimize on the assumption of no races–say, by hoisting the read of counter out of the loop and losing one thread’s entire hundred-thousand increments. Same source, symptom spectrum from “undetectable” to “half the work vanishes,” depending on the mood of the compiler and the silicon. Coincidentally harmless hardware does not make a well-defined program.
Example 4: one structure, a consequence spectrum from “forever invisible” to “occasional crash”
// race_lazyinit.cpp
#include <cstdio>
#include <thread>
struct Widget {
int id;
double ratio;
Widget() : id(42), ratio(3.14) {}
};
Widget* instance = nullptr;
Widget* get() {
if (!instance) { // check
instance = new Widget(); // act: a time window sits between the two
}
return instance;
}
void user() {
for (int i = 0; i < 1000; ++i) {
Widget* w = get();
if (w->id != 42) std::printf("corrupt!\n");
}
}
int main() {
std::thread a(user), b(user);
a.join(); b.join();
}
On most machines this code is “correct forever.” But its race structure (TOCTOU: check-then-act is not atomic) unfolds into a full spectrum of consequences:
- Mildest: both threads pass the check, each constructs a Widget, one leaks. Zero visible symptoms.
- Middle: on platforms that permit reordering, or under an aggressively optimizing compiler, the allocate-construct-publish sequence of
instance = new Widget()can be observed as “published before construction finished,” and a reader gets a half-built object. - Worst: combined with a later
deleteor cross-thread destruction, it decays into use-after-free–occasional crashes, Heisenbug-grade hard to debug.
Traditional debugging can’t even begin until the far right end of that spectrum–the physical event–occurs. TSan delivers the structural verdict at the far left end. That is the difference between “proving existence” and “waiting for occurrence”–the engineering cash-out of the Heisenbug piece’s Shift IV (reproduce the trigger conditions, not the bug).
TSan’s honest limits
Two things must be said plainly:
- TSan depends on execution coverage, not on consequences. It reports pairs of unordered conflicting accesses that actually happened during this run. If a code path never executed, TSan has nothing to say about it. The correct way to run TSan is with the full test suite, traffic replay, and load tests–maximize execution–not “ran it once, no races, ship it.”
- Both false positives and false negatives stem from the visibility of logical edges. Uninstrumented third-party code (assembly, hand-rolled atomics, custom synchronization primitives that bypass pthread) generates no vector-clock events, so TSan either reports a race that is actually synchronized (edge invisible) or misses a real one (edge falsely built). Custom synchronization needs explicit happens-before annotations (
ANNOTATE_HAPPENS_BEFORE/AFTER); reports you’ve confirmed benign go into a suppression file. Don’t “fix” it by turning TSan off.
V. UBSan: Turning the Compiler’s Assumptions Into Explicit Assertions
The mechanism: insert a check at the exact step where the assumption is about to be violated
The Debug/Release differences piece (Chinese) covered the root of optimization-divergence bugs: the C/C++ standard declares vast swaths of behavior undefined, and the optimizer is entitled to assume your program contains no UB and to transform code on that assumption. Loop vectorization, hoisting, dead-code elimination at -O2 all stand on it.
What UBSan does is, mechanically, disarmingly simple: immediately before the instruction where an assumption is about to be violated, insert a check; if the check fails, report. No shadow memory like ASan/TSan–just a few hundred assertions scattered through your code, each mapping to one clause of the standard:
- The statically decidable cases (constant scenarios) are caught by compile-time warnings anyway; instrumentation is mainly for cases whose values are only known at runtime;
- The usual set: signed overflow, out-of-range shifts, division by zero, null pointers, misalignment, invalid enum/bool values, array indexing out of bounds (
-fsanitize=bounds), bad polymorphic casts (vptr).
The verdict term is logical all the same: “is this shift exponent >= the bit width,” “does this addition stay within int‘s representable range.” Assertions written into the standard have nothing to do with how fast your program runs or how memory is arranged.
Example 5: x + 1 < x–an overflow check that vanishes into thin air
// ub_fold.cpp
#include <cstdio>
bool wraps(int x) {
return x + 1 < x; // "old-school overflow check": true if x+1 wraps below x
}
int main() {
std::printf("%d\n", wraps(2147483647)); // INT_MAX
}
Three runs for comparison (the first two verified on this machine while writing):
# MSVC: both /Od and /O2 print 1
cl /Od ub_fold.cpp && ub_fold # 1: hardware wraps in two's complement, INT_MAX+1 == INT_MIN < INT_MAX, "check works"
cl /O2 ub_fold.cpp && ub_fold # 1: this version doesn't fold it, the check "still works"
# clang / gcc: -O2 prints 0
clang++ -O2 ub_fold.cpp -o f2 && ./f2
# 0: compiler reasoning--x+1<x can only be true via signed overflow,
# overflow is UB, therefore assume it never happens
# -> expression is always false -> the function folds to return false
# UBSan intercepts at the moment the overflow happens
clang++ -O1 -g -fsanitize=undefined ub_fold.cpp -o fub && ./fub
# ub_fold.cpp:4:18: runtime error: signed integer overflow:
# 2147483647 + 1 cannot be represented in type 'int'
Clang’s optimizer (InstCombine) has an explicit transform: an x + 1 < x comparison on an nsw-flagged add is always false. This is not a compiler bug; it’s the compiler exercising the right the standard grants it–assume your program is UB-free, and simplify on that assumption.
Sit with how awkward the first two runs are: on MSVC the check “works, always has.” Move to clang/gcc at -O2 and it was legally deleted. Three compilers, one source file, three answers, all conforming. That is Family Three (optimization divergence) in its most vivid form: you test on your compiler; you ship to your users’ platforms–and the entire lifeline of “check overflow via wraparound” style code hangs on the compiler’s mercy. UBSan’s value comes into focus here: it doesn’t care how any particular optimizer disposes of this code; it reports exactly when “the operation that violates the standard” happens. By default it reports and continues (recover mode, so one run collects every violation); add -fno-sanitize-recover=undefined to make it stop on the spot.
Example 6: the out-of-range shift–hardware running interference for you
// ub_shift.cpp
#include <cstdio>
int main() {
int bits = 32; // shift count only known at runtime
std::printf("%d\n", 1 << bits); // x86 hardware takes the count mod 32
}
g++ -O1 -g -fsanitize=undefined ub_shift.cpp -o ub && ./ub
# ub_shift.cpp:6:26: runtime error: shift exponent 32 is too large for 32-bit type 'int'
Shifting a 32-bit int by 32 is UB, but x86’s shl quietly takes the count modulo 32, so 1 << 32 prints 1–“looks fine.” This is UB in its most insidious form: the hardware happens to be forgiving, so the program looks healthy. At -O2, the compiler is entitled to transform loops and vectorize on the assumption “shift exponents are always less than the width”–and the moment a runtime bits=32 punctures that assumption, every downstream transformation loses its footing.
Overhead and production-readiness: UBSan is the only family member that ships
Core integer checks cost on the order of 1-2% at runtime (heavy checks like vptr cost much more; enable selectively). More important is trap mode: -fsanitize-trap=undefined turns violations into CPU trap instructions (no runtime library linked, no formatted report), pushing the cost lower still. Android and Fuchsia both ship selected UBSan checks in trap mode in production, harvesting violation sites through the crash-reporting pipeline. A sharp division of labor versus ASan/TSan:
| ASan/TSan/MSan | UBSan (trap) | |
|---|---|---|
| Cost | 2-15x | ~1% |
| Report | full symbolized stacks | crash location only |
| Where it lives | CI / verification builds | can ship |
One last division of labor: UBSan covers ASan’s blind side. Signed overflow, null pointers, shifts–value-level UB–produce no memory-layout anomaly, so ASan can’t see them; the two stacked (-fsanitize=address,undefined is a legal combination) form something close to a complete memory-safety net.
VI. The Family at a Glance: How Many Builds Should a Project Keep
Lay the five tools out, each guarding its own detection territory:
| Tool | Detection domain | Cost (slow / memory) | Production-ready | Exclusivity | Compiler support |
|---|---|---|---|---|---|
| ASan | UAF, heap/stack/global overflows, double-free, (with LSan) leaks | ~2x / 2-3x | No | mutually exclusive with TSan, MSan | gcc, clang, MSVC (VS2019+) |
| MSan | uninitialized reads | ~3x / ~3x | No | exclusive with ASan, TSan; needs full instrumentation | clang only |
| TSan | data races, lock-order inversion (potential deadlocks) | 5-15x / 5-10x | No | exclusive with ASan, MSan | gcc, clang (not MSVC) |
| UBSan | value- and type-level UB | ~1-2% (lower in trap mode) | Yes (trap) | stackable with ASan | gcc, clang |
| LSan | leaks (checked at process exit) | nearly free | Yes | included with ASan by default | gcc, clang |
(HWASan uses hardware address tagging to shrink shadow memory to 1/16, at the price of probabilistic UAF detection; ARM only. Out of scope here.)
MSan’s “full instrumentation” deserves its own explanation–it’s another corollary of shadow memory: MSan gives every byte a “is it initialized” shadow bit; reading a poisoned bit fires a report. But any uninstrumented code (say, a statically linked third-party library) that writes memory doesn’t clear the shadow bit as a side effect–the memory it wrote “looks” uninitialized forever; conversely, garbage written by uninstrumented code is “laundered” because its shadow was never poisoned. So an MSan build must instrument every dependency too (including a self-built libc++), which makes it the highest-engineering-threshold member of the family.
The engineering conclusion follows naturally: a healthy C++ project maintains at least three verification builds–ASan for memory, UBSan for values, TSan for concurrency (plus an optional fourth: the fuzz build, Section VIII). They can’t be merged (mutually exclusive) and shouldn’t be (nobody can afford 2-15x overheads stacked).
VII. Blind Spots and False Positives: The Honest Limits of Probes
Tools have edges, and users need to know where they are–otherwise “all green” becomes a new form of self-deception.
ASan’s blind spots:
- Uninitialized reads (MSan’s turf), data races (TSan’s turf), pure logic errors (nobody’s turf).
- Intra-object overflow: a
char buf[16]member overflowing into the adjacentint xmember–the whole struct is “legitimate territory,” and the shadow says nothing. This is the most common “ASan was on, why didn’t it catch it?” scenario. Mitigations: heap-allocate hot buffers separately, or use the experimental field-padding option. - Container-internal overflow: writing into a
std::vector‘s[size, capacity)range is legal memory but a logical violation. libc++ (and newer libstdc++) ship annotations that catch it, given a matching standard library. - Custom allocators: memory pools that bypass
mallocare opaque to ASan; you must register poison regions manually via the__asan_poison_memory_regionfamily, or every in-pool overflow goes unreported.
TSan’s blind spots and false positives: see the end of Section IV–coverage dependence; uninstrumented libraries and custom synchronization causing false positives/negatives; suppressions suppress the report, not the race.
MSan’s threshold: no full instrumentation, no trustworthy results (Section VI).
UBSan’s boundary: the default set excludes unsigned wraparound (that’s well-defined behavior; -fsanitize=unsigned-integer-overflow exists as an opt-in lint); the vptr check needs RTTI and is expensive, usually weighed separately.
And one rule of thumb that binds all five: “no report” means precisely “no violation of any assertion I can check occurred in this execution.” Full stop. Probes are existence-proof tools, not absence-proof tools. Tape that sentence next to your CI’s green badge; it will save you a lot of retrospective regret.
VIII. Operationalizing: Wiring Probes Into CI and Production
Running a sanitizer by hand once is a demo. Making them work continuously is engineering. A copy-ready checklist:
1. The build matrix (three standing CI builds + one optional)
# ASan + UBSan (stackable)
g++ -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer app.cpp -o app_asan
# TSan (separate build; exclusive with ASan)
g++ -O1 -g -fsanitize=thread -fno-omit-frame-pointer app.cpp -pthread -o app_tsan
-fno-omit-frame-pointer stops stack unwinding from guessing; report quality improves immediately. Nightly: full test suite + real traffic replay. PRs: incremental cases only.
2. The runtime options that matter
ASAN_OPTIONS=detect_leaks=1:halt_on_error=0:abort_on_error=1:fast_unwind_on_malloc=0
TSAN_OPTIONS=halt_on_error=0:suppressions=tsan.supp
UBSAN_OPTIONS=print_stacktrace=1
The halt_on_error=0 + abort_on_error=1 combo: finish the run and collect every violation (don’t stop at the first), but exit nonzero so CI goes red. fast_unwind_on_malloc=0 makes allocation stacks complete.
3. The reaction with fuzzing
Probes solve “how violations become visible”; fuzzing solves “how to make violations happen.” Under -fsanitize=fuzzer,address, coverage-guided fuzzing keeps generating new inputs, and any input that triggers a memory violation detonates right at the offending instruction, with the full input attached. The two together are the complete exposure machine–OSS-Fuzz has used exactly this combo to dig tens of thousands of memory and UB defects out of open source, the overwhelming majority intercepted in CI before they ever fired.
4. The production-side division of labor
- UBSan trap mode rides along with release builds (Section V); the crash-reporting pipeline doubles as the violation-collection pipeline.
- LSan is nearly free; run it as a routine exit check.
- ASan/TSan don’t go to production–unless you run a shadow-traffic cluster built specifically for verification builds.
5. The baton pass: when probes can’t catch it
When a bug fires only under real production load or specific data–execution paths your verification suite never covers–probes yield. Switch to recording: on Windows, WinDbg TTD; on Linux, rr; record one real failure and replay offline (see “Reverse Debugging: Principles and Use Cases,” not yet published). Or wait for the body: take a core dump and analyze offline (see “Dump Files: Principles, Types, Analysis, and Linux Practice,” not yet published). The three weapons’ division of labor in one line: probes check structure, recordings restore timing, autopsies examine the scene.
6. A closing note for the AI era
The value of all this is being re-priced upward. AI-generated code has one systemic property: an unusually high fraction of it looks correct, and “looks correct” is exactly what lets structural memory violations, overflows, and UB slip past human review (Bugs Written by AI Look Like Correct Code). As generated-code throughput rises, the cost curve of “line-by-line human review as the only quality gate” must break. Probes are the only structural verifier that scales at the same order of cost–they don’t read intent, they verify assertions, which makes them precisely the reviewer AI code needs. Five years ago sanitizers were an advanced technique. Today they are infrastructure.
IX. Closing: From “Catching a Bug” to “Proving One Exists”
Look back at the road this article traveled.
In the opening experiment, a data race ran “quietly correct” for three years. Traditional debugging’s posture toward it is waiting: wait for a physical consequence–a crash, a corruption–to become visible, then work backward from it. The Heisenbug piece already argued why that posture fails: your observation rewrites the result; more effort moves you further away. Sanitizers adopt a fundamentally different posture: don’t wait for the physical event; verify the logical assertion directly.
The three weapons fall into place within this frame:
- The core dump is the autopsy–wait for the physical event (death), then dissect the scene;
- rr/TTD is the video recording–freeze one physical run’s timing, replay it endlessly afterward;
- The sanitizer is the surveillance probe planted inside the system–it doesn’t wait for events; it verifies constraints.
They share one criterion–Section II’s foundation: the assertion your detection relies on must not depend on the variable you disturbed. Breakpoints backfire because their verdict variable (the timing window) is exactly their perturbation variable. Probes stand because they rewrote the verdict from “physical coincidence” (witnessing a window, crossing a page by luck, landing on the right neighbor) into “logical violation” (stepping into poison, a missing partial order, breaching a standard clause). Physical coincidences take luck; logical assertions are present at every execution.
One final layer, back to epistemology. This series keeps repeating one line: debugging is a conversation with a system that changes because of you, and what you read is always a projection under your chosen mode of observation. Sanitizers don’t cancel that line–the TSan build’s system still isn’t the native system; the ASan build’s heap still isn’t the native heap. What they do is something smarter: since every observation projects, design a quantity that is invariant under projection. However the timing is slowed down or the layout rearranged, “is this access inside the object’s bounds” and “are these two accesses unordered” keep the same answer. Engineering calls this deterministic detection; epistemology calls it building a wall between the variable you depend on and the variable you disturb.
The next layer is already coming into view: the happens-before partial order TSan uses to judge concurrency is precisely the ordering language Lamport invented for distributed systems–a shared-memory multithreaded program is, logically, a distributed system; and “there is no privileged observer; every observer carries their own clock and their own causal cone” opens onto a much larger map. That is another article’s business.
What you can take away today is three sentences:
Whether the program is sick and whether you have seen symptoms are two independent questions.
Probes don’t cure the disease; they turn “running sick” from a stroke of luck into a readable report.
Non-invasive doesn’t mean non-intruding. It means the evidence doesn’t depend on the thing you intruded on.
Appendix A: Symptom Fingerprint -> Probe Quick Reference
(The four root-cause families are laid out in Section III of the Heisenbug piece.)
| Symptom fingerprint | Family | First-choice probe | Report keywords |
|---|---|---|---|
| Multithreaded, never reproduces under single-step, improves with logging | One (timing) | TSan | data race / lock-order-inversion |
Crashes only in release, reads 0xCD/0xDD pattern garbage | Two (initialization) | ASan (UAF/overflow), MSan (uninit reads) | heap-use-after-free / use-of-uninitialized-value |
Crashes at -O2 only, innocent-looking stack | Three (optimization/UB) | UBSan | signed integer overflow / shift exponent / misaligned |
| Crash stack drifts every time; changes across machines/builds | Four (layout) | ASan | heap-buffer-overflow / stack-buffer-overflow / global-buffer-overflow |
| Suspected leaks | – | LSan (included in ASan) | detected memory leaks |
| All probes green, still intermittent in production | – | Abandon probes; record or autopsy | TTD / rr / core dump |
Appendix B: Compile Flags and Runtime Options Cheat Sheet
# ASan + UBSan (legal to stack)
g++ -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer app.cpp -o app_au
# TSan (separate build)
g++ -O1 -g -fsanitize=thread -pthread -fno-omit-frame-pointer app.cpp -o app_t
# MSan (needs full instrumentation incl. dependencies; clang only)
clang++ -O1 -g -fsanitize=memory -fno-omit-frame-pointer app.cpp -o app_m
# UBSan production trap mode (no runtime library)
g++ -O2 -fsanitize=undefined -fsanitize-trap=undefined app.cpp -o app_prod
# fuzz build (coverage-guided + ASan)
clang++ -O1 -g -fsanitize=fuzzer,address app.cpp -o app_fuzz
# Common runtime options
ASAN_OPTIONS="detect_leaks=1:halt_on_error=0:abort_on_error=1:fast_unwind_on_malloc=0"
TSAN_OPTIONS="halt_on_error=0:suppressions=tsan.supp"
UBSAN_OPTIONS="print_stacktrace=1"
# MSVC (VS 2019 16.9+)
cl /EHsc /O1 /Zi /fsanitize=address app.cpp
Note: MSVC supports ASan only; for TSan/MSan use gcc/clang (Linux/macOS home turf).
Related Posts
- The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging–the hub prequel: the three weapons and the methodology of decoupling measurement from system
- Bugs Written by AI Look Like Correct Code: An Epistemic Rewrite of Debugging–why the AI era turns probes from technique into infrastructure
- Debug vs. Release in Visual Studio, In Depth (Chinese)–the mechanistic prequel to Family Three (optimization divergence)
Leave a Reply