This far into the debugging series, there are now three sentences:
A crash lies in space. A hang lies in time. Slowness lies in statistics.
The first was about heap corruption: the crash site is not the crime scene. The second was about hangs: standing still is a reading, not a mechanism. This one completes the third: slowness is not a number, it’s a distribution. Put the three together and they say the same thing — the moment you treat an observed phenomenon as a property of the system, diagnosis turns into guesswork.
Boundaries first. Every number below is a real recording from my own machines: nothing prettified, no cherry-picked “good run.”
- Windows 11 (10.0.26300), MSVC 19.51.36260 x64, 8 vCPU;
- WSL2 Ubuntu 24.04.1, gcc 13.3.0, kernel 6.18.33.2, 8 vCPU.
The one thing that is not a recording is the main case in Section VII. It is a synthetic illustration whose parameters come from the M/M/1 formulas at the same order of magnitude as the measurements here, built to demonstrate the attribution path — not a postmortem of a real incident. I’ll say so again at the top of that case.
I. First, Take Apart a Premise: Performance Is Not a Property, It’s a Distribution
When an engineer says “this endpoint takes 20ms,” an adjective is usually missing: which statistic? The mean? The median? p99? The omission lets a team assume everyone means the same thing, and they usually don’t.
I wrote a minimal experiment: three pieces of code, each simulating 2,000 request service times. A and B share an expected mean of 6ms but have completely different shapes; C has a lower mean and a much heavier tail.
- A, stable: 6ms every time;
- B, spiky: 98% of requests take 3ms, 2% take 151.5ms, mean still about 6ms;
- C, long-tailed: 90% at 2ms, 9% at 20ms, 1% at 100ms.
Windows, measured:
hist: same mean, different distribution (all values in ms)
A-stable n=2000 mean= 6.03 p50= 6.00 p90= 6.00 p99= 6.44 p999= 14.36 max= 21.39
B-spiky n=2000 mean= 5.98 p50= 3.00 p90= 3.00 p99= 151.50 p999= 151.50 max= 151.51
C-longtail n=2000 mean= 4.41 p50= 2.00 p90= 20.00 p99= 20.00 p999= 100.00 max= 100.00
Linux, measured:
A-stable n=2000 mean= 6.03 p50= 6.00 p90= 6.00 p99= 6.55 p999= 10.41 max= 16.81
B-spiky n=2000 mean= 6.04 p50= 3.00 p90= 3.02 p99= 151.50 p999= 151.51 max= 152.35
C-longtail n=2000 mean= 4.43 p50= 2.00 p90= 20.00 p99= 20.06 p999= 100.00 max= 100.00
Start with A and B. Their means are 6.03 and 5.98 — a 0.8% gap, which on any dashboard reads as “no change.” And yet:
- B’s median is 3ms, twice as fast as A’s;
- B’s p99 is 151.5ms, 23 times worse than A’s;
- B’s max is essentially equal to its p99, which tells you this is not occasional jitter but a second path that exists all the time.
Look only at the mean and you’ll conclude the two systems are about the same. Look only at p50 and you’ll conclude B is better. Only p99 tells you that 2% of B’s users wait 151ms. The mean spreads that 2% disaster into a 3ms increment and makes it look like “a bit slower.”
Now C. It has the lowest mean of the three (4.41ms) and is simultaneously the one you can least afford to make promises about: 90% of requests land inside 20ms, 1% take 100ms, and p999 jumps straight to 100ms. The mean is low because most requests are fast; p999 is high because the slow path is real. Both are true at once, and “4.4ms on average” conceals both.
So the first question isn’t “is it fast,” it’s what shape is the distribution. Three shapes cover most of what you’ll meet:
- Shift: the whole distribution moves right; p50 and p99 degrade in proportion. Usually the resource itself got slower — down-clocked CPU, a slower downstream, higher network RTT.
- Tail: p50 holds, p99 degrades. Usually a slow path, queueing, retries, GC, cache stampede — or the observer itself.
- Bimodal: a dip in the middle, or a plateau between p90 and p99. Usually two code paths (hit/miss, hot/cold, sync/async) blended into one metric.
Establish the shape before you discuss optimization. Get the shape wrong and optimization is guesswork.
II. Three Physical Shapes of Slow: Computing, Waiting, Queueing
“Slow” is not one physical state. It’s at least three:
- On-CPU: the thread is executing instructions on a core;
- Off-CPU: the thread is blocked — on I/O, a lock, a downstream, a timer;
- Queued: the request has arrived but processing hasn’t started; it’s sitting in a queue.
All three can burn identical wall-clock time and demand completely different fixes. One ratio separates them: CPU time consumed by the process (or thread), divided by wall-clock time.
Same code, three variants:
- A: 100 computations, 4ms each;
- B: 40 waits, 10ms each;
- C: 100 computations (4ms each), plus a 1ms wait after each one.
Windows, measured:
A-on-CPU wall= 400.0ms user= 390.6ms sys= 15.6ms cpu_total= 406.2ms cpu/wall=101.6%
B-off-CPU wall= 630.4ms user= 0.0ms sys= 0.0ms cpu_total= 0.0ms cpu/wall= 0.0%
C-cpu-plus-wait wall= 1561.0ms user= 296.9ms sys= 0.0ms cpu_total= 296.9ms cpu/wall= 19.0%
Linux, measured:
A-on-CPU wall= 792.9ms user= 793.0ms sys= 0.0ms cpu_total= 793.0ms cpu/wall=100.0%
B-off-CPU wall= 406.5ms user= 1.5ms sys= 0.0ms cpu_total= 1.5ms cpu/wall= 0.4%
C-cpu-plus-wait wall= 793.3ms user= 682.3ms sys= 0.0ms cpu_total= 682.3ms cpu/wall= 86.0%
A few things here deserve a pause.
First, B’s wall clock is 630ms and its CPU time is zero. It really did “take 630ms,” and not one line of code was computing. Point a CPU profiler at it and you get a blank profile — not because the sampler missed anything, but because there was nothing on the CPU to sample. A flat flame graph doesn’t mean no time was spent; it may simply mean the time wasn’t on the CPU.
Second, C’s wall clock is 3.9x A’s, but CPU time didn’t grow with it: A burned 406ms, C only 297ms (same order of magnitude), while wall clock went from 400ms to 1,561ms. In other words, the slowdown didn’t come from more computation; it came from waiting inserted between computations. Plan capacity by CPU time and you’ll conclude everything is fine; hold an SLO against wall clock and you’re already in breach.
Third, the same C code shows CPU/wall of 19% on Windows and 86% on Linux. The reason isn’t the code, it’s sleep granularity: a 1ms sleep on Windows is really 15.6ms; on Linux it’s 1.1ms (measured in Section IV). The same code running on two different physical systems — that’s not a metaphor, it’s literal.
Two caveats. The A phase running slightly above 100% CPU/wall on Windows is normal skew from timer overhead and the way CPU time is accounted. The A phase overshooting its 400ms design on Linux (793ms) is WSL2 scheduling jitter under host load, but CPU time still tracks wall clock, so the ratio conclusion stands.
Each shape also has its own tooling:
| Shape | Direct evidence | Usual tools | What to fix |
|---|---|---|---|
| Computing | High CPU time, hotspot in samples | perf record, VTune, ETW/WPA, Visual Studio diagnostic tools | Algorithm, instructions, cache, parallelism |
| Waiting | Low CPU time, thread blocked | perf sched, offcputime, ETW wait analysis, strace, lock analysis | Fewer waits, batching, async, timeout and retry policy |
| Queueing | CPU not high, queue depth / wait time high | Thread-pool metrics, connection-pool metrics, queueing models | More parallelism, load shaping, rate limiting, scale out |
A common misread: seeing CPU at 45% and concluding “there’s plenty of headroom.” Idle CPU only says nothing is computing; it says nothing about what is waiting or queueing. One global lock, one connection pool, one partition can push p99 to 800ms while 3 of your 8 cores are busy. Section V gives the formulas; Section VII gives the case.
III. Queueing: Nonlinearity Is Not a Bug, It’s Math
The single-server queue is the smallest useful model: Poisson arrivals at rate λ, exponentially distributed service at rate μ, one server. Utilization:
ρ = λ / μ
Expected queueing delay:
E[Wq] = ρ / (μ(1 − ρ))
There is also an atom of probability at zero wait (1 − ρ); only the continuous part is exponential. So p99 isn’t a tidy 4.6x the mean. It is:
p99 = ln(100ρ) / (μ − λ)
I wrote a discrete-event simulation with one million customers and checked it against those formulas. Both platforms agree digit for digit (deterministic RNG); the table is the Windows recording, and Linux is identical:
rho mean_wait p99_wait theory_mean theory_p99
0.50 1.00ms 7.82ms 1.00ms 7.82ms
0.80 3.96ms 21.66ms 4.00ms 21.91ms
0.90 9.07ms 45.41ms 9.00ms 45.00ms
0.95 19.56ms 97.33ms 19.00ms 91.08ms
0.98 53.48ms 262.95ms 49.00ms 229.25ms
Look at the means first: ρ goes from 0.5 to 0.98 and mean queueing time goes from 1ms to 53ms — 50x. Now p99: 7.8ms to 263ms — 34x. And ρ itself barely doubled.
At ρ=0.98 the simulated p99 runs 15% above theory (263ms versus 229ms). That’s sampling error, not a broken formula: with a million samples only about ten thousand sit above p99, and extreme quantiles are inherently unstable. Which is also why the sample-size question in Section VI matters.
Isolating “queueing time / service time” makes it more vivid:
| ρ | ρ/(1−ρ) | What it means |
|---|---|---|
| 0.50 | 1.0 | Average queueing time equals one service time |
| 0.70 | 2.3 | Queueing time is 2.3x service time |
| 0.90 | 9.0 | 9x |
| 0.95 | 19.0 | 19x |
| 0.98 | 49.0 | 49x |
This is where the “70% rule” comes from: past 70% utilization, waiting time starts to diverge as 1/(1−ρ). Note diverge, not grow linearly. The most common capacity-planning mistake is reading “20% CPU left” as “we can add 20% more traffic.” Add 20% traffic on top of ρ=0.8 and ρ lands at 0.96; p99 doesn’t rise by 20%, it rises by roughly 5x.
The third variable is variance. M/M/1 assumes exponential service times; real service times are more variable (long tails, retries, GC), and then you reach for Kingman’s approximation:
E[Wq] ≈ (ρ / (1 − ρ)) × ((c_a² + c_s²) / 2) × E[S]
where c_a and c_s are the coefficients of variation of the interarrival and service times. This explains a puzzle people hit constantly: a service that averages 1ms can have a p99 of 100ms. p99 is not set by the mean alone; utilization and variance decide it together. Shave the mean service time without touching variance and p99 may not move at all.
So p99 is a property of the system, not of a function. Three things set it: utilization, variance, and capacity. Treating p99 as a synonym for “the code is slow” is the single most common performance misjudgment there is.
IV. Observation: What You Measure Is Not the System You Started With
Time to put the ruler itself on the table.
Observation has at least three effects:
- Instrumentation cost: every clock read, log write, and counter increment costs time;
- Sampling bias: your sampling rate decides what you can see — and what you can’t;
- The observer consumes resources: the monitoring thread takes CPU, fights over cache lines, and writes logs that eat I/O.
Instrumentation cost first. Fixed workload (50 million integer operations), four granularities:
- no instrumentation;
- one sample every 1,000 iterations;
- one every 100;
- every iteration.
Windows, measured:
clock-read cost = 24.3 ns/call (20000000 calls in 486.7 ms)
no-instrumentation median= 81.9ms overhead= +0.0%
clock-every-1000 median= 85.0ms overhead= +3.8%
clock-every-100 median= 103.3ms overhead= +26.1%
clock-every-1 median= 1232.5ms overhead=+1404.9%
Linux, measured:
clock-read cost = 47.1 ns/call (20000000 calls in 941.8 ms)
no-instrumentation median= 115.2ms overhead= +0.0%
clock-every-1000 median= 125.9ms overhead= +9.3%
clock-every-100 median= 146.5ms overhead= +27.1%
clock-every-1 median= 2268.1ms overhead=+1868.2%
The number that matters here isn’t 1,404%; it’s where it comes from: a unit mismatch. The unit of work being observed costs about 1.6ns (81.9ms / 50 million), while one clock read costs 24ns on Windows or 47ns on Linux. The observer’s granularity is an order of magnitude coarser than the thing observed, so “observe every unit” has to inflate the system tenfold.
On a real system, redo that arithmetic in real units. Say two spans per request at 30ns each:
- 10k QPS: 2 × 30ns × 10⁴ = 0.6ms/s, about 0.06% of a CPU — forget about it;
- 1M QPS: 60ms/s, about 6% of a CPU — now it’s on the books;
- and if a “span” becomes “one log line per request” (formatting plus writing, microseconds to milliseconds), 10k QPS turns into 10ms to 10s of overhead per second, and you have changed the system.
So observability is neither free nor better the finer it gets. Observation budget = cost per span × spans per second. That’s design-time arithmetic, not a guess you make after launch.
The second effect is sampling bias. A 1kHz sampling profiler takes a stack every 1ms. It sees hotspots lasting tens of milliseconds, but:
- spikes shorter than 1ms can slip through entirely;
- threads blocked in I/O, locks, or timers have no CPU stack to sample;
- which thread gets sampled depends on scheduling; it isn’t uniform.
That’s how “the flame graph is flat” and “p99 is terrible” can both be true. A flame graph answers “where is the CPU burning,” not “where is the time waiting.”
The third effect is the observer consuming resources. The monitoring thread occupies a core; if the counters it reads share a cache line with application data, every read knocks that line from Modified back to Shared, and the next write on the application side has to re-acquire it exclusively. At low scrape rates this is noise; at high rates it’s real throughput loss. It is the same species as instrumentation cost: the observer’s actions enter the causal chain of the system it observes.
Last, timer granularity. One line of “sleep 1ms,” measured on both platforms:
Windows:
default n=300 mean= 15.57ms p50= 15.60ms p90= 16.07ms p99= 16.50ms max= 16.77ms <2ms: 0.0%
timer-1ms n=300 mean= 1.80ms p50= 1.96ms p90= 2.19ms p99= 2.94ms max= 3.50ms <2ms: 65.3%
Linux:
default n=300 mean= 1.12ms p50= 1.12ms p90= 1.16ms p99= 1.22ms max= 1.30ms <2ms:100.0%
Windows’ default sleep granularity is 15.6ms, so you ask for 1ms and get 15.6ms; Linux delivers 1.1ms. That difference alone rewrites the CPU share in the Section II C experiment (19% versus 86%), and it rewrites any “1ms timeout” logic you own.
I also tried explicitly requesting 1ms timer resolution (timeBeginPeriod). On the same machine it sometimes dropped p50 to 1.96ms and sometimes stayed at 15.5ms. Which is to say, even calibrating the ruler is not stable. That isn’t an API bug; it’s a reminder that the measurement environment is a system that changes, not a constant backdrop.
The conclusion is plain: measure the ruler before you measure the system. Record at least three things — clock resolution, cost per observation, and the resources the observer itself consumes. Otherwise you are reporting the joint distribution of “system + ruler” and calling it the system.
V. Unit Mismatch: Three Kinds of Time and Three Formulas
Three kinds of time get mixed up constantly in performance discussions:
- Wall-clock time: what the user experiences;
- CPU time (user + sys): what the process actually took from the processor;
- Wait time (wall − CPU): blocked, queued, sleeping.
Then there are two kinds of time you can’t see: steal time under virtualization (the VM preempted by the host) and CPU throttling in containers (suspended the moment the quota is spent). Both present exactly like off-CPU waiting — wall clock up, CPU flat.
Three formulas worth memorizing:
- Little’s Law: L = λW. Concurrency = arrival rate × time in system. 100k QPS at a 100ms average means 10,000 requests in flight; if your thread pool has 200 threads, queueing is a certainty.
- CPU demand: CPU time = QPS × CPU time per request. 1,000 QPS at 2ms of CPU each is two saturated cores.
- Capacity: ρ = λ / (c × μ), where c is parallelism. Target ρ ≤ 0.7; the more variance, the lower the target.
Three common mistakes:
- Explaining latency with CPU utilization. 45% CPU says “little is computing,” not “nothing is queueing.”
- Sizing concurrency from the mean. W in Little’s Law is average time in system, but your SLO is about p99. Concurrency computed from the mean systematically understates what the tail periods actually occupy.
- Budgeting CPU from wall clock. A request with 100ms of wall clock and 5ms of CPU, sized at 100ms per request, is 20x the hardware you need.
VI. The Five-Step Differential Diagnosis
Compress all of the above into a procedure you can actually run.
Step 1: Fix the denominator. Slower than what? Is the SLO “p99 under 200ms,” or “slower than yesterday”? Different denominators can flip the conclusion. Write the comparison target down: the SLO, the baseline build, peer requests, a control on the same host.
Step 2: Draw the distribution. At minimum p50, p90, p99, p999, max — ideally a histogram. Decide whether it’s a shift, a tail, or bimodal. Sample size matters: a stable p99 wants on the order of a hundred times 1/(1−p) samples; 10k samples is already marginal for p99, and p999 needs north of 100k.
Step 3: Split the three states. Take CPU time and wall clock over the same window and compute CPU/wall. Above 80% is mostly computing; below 20% is mostly waiting; in between is mixed. Then split “waiting” into queueing, locks, I/O, downstream, and timers.
Step 4: Attribute to a shared resource. A single instance’s bottleneck is rarely “total CPU” — it’s one lock, one connection pool, one partition, one downstream. Find that μ, compute its ρ, not the machine’s ρ.
Step 5: Differential test. Change one variable at a time and watch which segment of the distribution moves. Turn off the observer, double the connection pool, drop the sampling rate tenfold, halve the traffic — one at a time, recording how p50 and p99 move. If p50 holds and p99 moves, the problem is in the tail; if both move in proportion, the problem is global.
Reading table:
| Symptom | Most likely mechanism | Next move |
|---|---|---|
| p50 and p99 rise in proportion | The whole resource got slower | Check CPU frequency, downstream RTT, network, disk |
| p50 flat, p99 up | Queueing, a slow path, retries, GC | Compute ρ; read the trace of slow requests; count retries |
| p99 high but CPU low | Off-CPU waiting or queueing | Check locks, connection pools, I/O, downstream; look at wait time |
| Mean down, p99 up | Fast path faster, slow path slower | Measure the two paths separately; don’t blend them into one metric |
| Slower once monitoring is on | Observer effect | Compute the observation budget; downsample; check cache lines and logging |
| Only at peak | Utilization has entered the nonlinear zone | Estimate with ρ and Kingman; scale out or rate-limit ahead of time |
Five common misjudgments:
- “CPU isn’t high, so it’s not a performance problem” — CPU only covers computing.
- “The mean didn’t change, so nothing regressed” — the mean can flatten a 2% disaster.
- “No hotspot in the flame graph, so there’s nothing to optimize” — the hotspot may not be on the CPU.
- “Just add machines” — if the bottleneck is one partition or one lock, more machines only queue more requests at the same μ.
- “Load testing passed” — load tests rarely match production’s ρ, variance, and observation overhead.
VII. Main Case: The p99 Incident Where “CPU Was Only 45%”
One more time: what follows is a synthesized case. Parameters follow the M/M/1 formulas at the same order of magnitude as the measurements above; it demonstrates the attribution path, not a postmortem.
An order service: 4 instances, 8 cores each. Every request passes through a serialized resource — a global statistics lock plus a single connection. That segment’s throughput is μ = 500/s. Pre-release traffic per instance is λ = 470/s.
Before the release:
- ρ = 470 / 500 = 0.94;
- mean queueing time = ρ / (μ(1−ρ)) = 0.94 / (500 × 0.06) = 31ms;
- p99 queueing time = ln(100 × 0.94) / (500 × 0.06) = 151ms;
- add 20ms of base processing: p50 about 20ms, mean about 51ms, p99 about 171ms.
The release adds a metrics agent that scrapes statistics every 100ms, holding that same lock for 4.8ms per scrape. Its share of capacity is 48ms per second — 4.8%.
Intuition says: 4.8%, how bad can it be?
Do the arithmetic:
- effective capacity of the serialized segment: μ_eff = 500 × (1 − 4.8%) ≈ 476/s;
- ρ = 470 / 476 = 0.987;
- mean queueing time = 0.987 / (476 × 0.013) = 165ms;
- p99 queueing time = ln(98.7) / (476 × 0.013) ≈ 765ms;
- add 20ms of base processing: p50 still 20ms, mean about 185ms, p99 about 785ms.
So the dashboard shows a very familiar combination: p50 unchanged, mean up 3.6x, p99 from 171ms to 785ms — and total CPU at 45%.
Why is CPU only 45%? Because the bottleneck isn’t eight cores, it’s that one serialized segment. The other seven cores are waiting on it. CPU utilization measures computing, not queueing.
Run the five steps:
- Fix the denominator: the SLO is p99 < 200ms; post-release it’s 785ms — a clear breach.
- Draw the distribution: bimodal, p50 at 20ms and p99 at 785ms, so this isn’t global slowdown.
- Split the three states: CPU/wall is only 12%, mostly waiting — and the waiting concentrates on the serialized segment.
- Attribute: compute ρ for the serialized segment; it moved from 0.94 to 0.987. The agent’s 4.8% is the trigger.
- Differential test:
– stop the agent: ρ back to 0.94, p99 back to 171ms;
– connection pool 1 → 2 (μ doubles to 1,000): ρ = 0.47, p99 queueing time drops to about 7ms (about 27ms total);
– loosen the agent’s scrape from 100ms to 1s: occupancy drops to 0.48%, ρ = 0.945, p99 back to about 185ms — the same order of magnitude as baseline.
The postmortem attribution comes out roughly: queueing 70%, observer 20%, and the remaining 10% extra jitter from the release. Of those three numbers, only 10% is “the code got slower.” The other 90% is the system amplifying a perturbation because utilization is high.
The point of this case is not “agents are evil.” It’s this: in a system running near ρ = 1, a few percentage points of extra occupancy can be amplified into multiples of p99. The observer isn’t standing outside the system looking in; it’s inside the causal chain.
VIII. Prevention: Make Performance a Gate
Performance isn’t something you optimize into existence; it’s something you defend. Here is a discipline you can adopt as-is.
- SLOs in quantiles only. Not “average latency under 100ms” but “p99 under 200ms, p999 under 500ms.” A mean can support a case; it can’t make a promise.
- Reports must carry the distribution. p50/p90/p99/max at minimum, ideally a histogram. Mean-only data cannot distinguish a shift from a tail from bimodality.
- Enough samples. Tens of thousands for p99, hundreds of thousands for p999. When you don’t have them, report an interval, not a point.
- Rotate your benchmarks. Five runs per configuration, rotating order, take the median; report min and max. A fixed order lets clock ramp and thermal drift contaminate the result.
- Plan capacity by ρ, not by the mean. Target ρ ≤ 0.7, lower for high-variance services. Before scaling out, establish whether the bottleneck is total capacity or a single partition.
- Compute the observation budget. Cost per span × QPS × spans per request; sample logs by level; keep high-frequency counters off shared cache lines.
- Gate regressions on p99 plus an absolute threshold. For example: “p99 up 20% versus baseline and above 200ms absolute, three consecutive runs.” A single noisy run should never wake anyone up.
- Record the environment fingerprint. OS build, compiler version, CPU model, virtualization layer, timer resolution, power policy. Without those, “can’t reproduce” is just a sentence.
Minimal checklist:
- [ ] For this measurement, what is the clock resolution?
- [ ] What does one span cost? How many spans per second?
- [ ] How much CPU does the observer take, and which cache lines does it fight for?
- [ ] Is the sample size enough to support p99 / p999?
- [ ] What is CPU/wall? Is the bottleneck computing, waiting, or queueing?
- [ ] What is ρ? Is the target 0.7, or lower?
- [ ] How large is the variance? Will retries amplify it?
- [ ] Can the conclusion be reproduced by changing one variable at a time?
IX. Boundaries: When Distribution Thinking Doesn’t Apply
Distribution thinking isn’t universal. At least four cases need care:
- One-off events: batch jobs, cold starts, migrations. There’s no distribution to speak of, only a timeline.
- Hard real-time systems: what matters is worst-case execution time (WCET), not quantiles. A beautiful p99 still fails if one deadline is missed.
- Incomparable distributions: p99s from different loads, different code paths, or different data distributions don’t belong in the same comparison. Confirm you’re looking at the same class of request first.
- Observation is the business: audit logs, billing records, security events. These can’t be sampled away for performance; the only move is a cheaper write path.
Queueing models also come with premises: Poisson arrivals, exponential service, a single server. Real systems rarely satisfy them. But their value isn’t precise prediction, it’s direction and magnitude: as utilization approaches 1, waiting time must diverge nonlinearly; the more variance, the earlier it diverges. That conclusion doesn’t depend on the specific distribution.
X. Closing: Optimization Isn’t Making Code Faster, It’s Making the Distribution Narrower
Back to those three sentences.
A crash lies in space, because the crash site points at the victim, not the killer. A hang lies in time, because “not moving” is a reading, not a mechanism. Slowness lies in statistics, because the mean, the median, and p99 describe three different systems, and you usually only asked about one of them.
The goal of performance work isn’t to push down some particular number; it’s to make the distribution narrower and more predictable:
- eliminating a 2% spike of 151ms is worth more than shaving the mean from 6ms to 5ms;
- dropping ρ from 0.98 to 0.7 beats buying a faster machine;
- moving observation from “per request” to “per thousand requests” is cheaper than buying two more cores;
- cutting waiting from 600ms to 60ms improves the experience more than cutting computation from 400ms to 350ms.
One last line, for everyone who does performance triage:
You think you’re measuring the system. You’re measuring the system plus your method of measuring it. Measure the method first; then we can talk about the system.
Appendix A: Percentile Interpretation Cheat Sheet
| Observation | Reading | First move |
|---|---|---|
| p50 up, p99 up in proportion | Global shift | Check whether the resource got slower overall: frequency, downstream, network, disk |
| p50 steady, p99 up | Tail problem | Check queueing, slow paths, retries, GC, the observer |
| p50 up, p99 steady | Fast path got slower | Check common preludes: auth, serialization, connection setup |
| Mean down, p99 up | Two paths diverging | Measure separately; don’t blend them into one metric |
| p99 high, CPU low | Off-CPU or queueing | Check locks, connection pools, I/O, downstream, queues |
| Only at peak | Utilization in the nonlinear zone | Compute ρ; rate-limit, scale out, reduce variance |
| Appears once monitoring is on | Observer effect | Compute the observation budget; downsample; isolate cache lines |
| Behaves differently on two platforms | Environment difference | Check timer granularity, scheduling, virtualization, compiler flags |
Appendix B: Measurement Commands, Side by Side
| Purpose | Windows | Linux |
|---|---|---|
| Monotonic clock | QueryPerformanceCounter | clock_gettime(CLOCK_MONOTONIC) |
| Process CPU time | GetProcessTimes | getrusage(RUSAGE_SELF) |
| Thread CPU time | GetThreadTimes | clock_gettime(CLOCK_THREAD_CPUTIME_ID) |
| Sleep | Sleep | nanosleep |
| High-resolution timer | CreateWaitableTimerEx(HIGH_RESOLUTION) | timerfd / clock_nanosleep |
| CPU sampling | ETW + WPA, PerfView, VTune | perf record / perf report |
| Scheduling and waits | ETW wait analysis, WPA | perf sched, offcputime |
| System calls | Process Monitor | strace |
| Lock analysis | ETW, Visual Studio parallel stacks | perf lock, gdb, TSan |
| Container quotas | Job Object / processor groups | cgroup v2 cpu.max |
Appendix C: Reproduction Environment and Scripts
Sources under scripts/latency/:
bench_common.h cross-platform clock, CPU time, sleep, percentiles
hist.c same mean, three distributions
queue.c M/M/1 simulation checked against theory
oncpu.c computing, waiting, and mixed
observer.c observation granularity and instrumentation cost
sleeptest.c timer granularity
build_win.bat MSVC build (/O2 /std:c11 /utf-8)
build_linux.sh gcc build (-O2 -std=c11)
Windows:
cd scripts\latency
build_win.bat
hist.exe & queue.exe & oncpu.exe & observer.exe & sleeptest.exe
Linux:
cd scripts/latency
./build_linux.sh
./hist && ./queue && ./oncpu && ./observer && ./sleeptest
The minimal measurement skeleton (equivalent to bench_common.h):
#ifdef _WIN32
static double now_us(void) {
static double freq = 0.0;
if (freq == 0.0) { LARGE_INTEGER f; QueryPerformanceFrequency(&f); freq = (double)f.QuadPart; }
LARGE_INTEGER c; QueryPerformanceCounter(&c);
return (double)c.QuadPart * 1e6 / freq;
}
#else
static double now_us(void) {
struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
return (double)ts.tv_sec * 1e6 + (double)ts.tv_nsec / 1e3;
}
#endif
Two notes:
- On Windows, read
QueryPerformanceFrequencyonce and cache it; otherwise every call adds two more API calls and inflates the measurement overhead (the first draft of this article walked into exactly that). - Percentiles use linear interpolation over sorted samples; when the sample size is thin, p999 is a trend, not a promise.
Appendix D: Environment Fingerprint
Windows: 10.0.26300 (Windows 11)
Compiler: Microsoft (R) C/C++ Optimizing Compiler 19.51.36260 x64
Linux: Ubuntu 24.04.1 LTS, kernel 6.18.33.2-microsoft-standard-WSL2
Compiler: gcc 13.3.0
CPU: 8 vCPU (WSL2)
Timer: Windows default ~15.6ms; Linux ~1.1ms
Related Posts
- The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging – how observation changes the system; the same problem, one level up
- Non-Invasive Probes: How ASan/TSan/UBSan Expose Bugs Before They Ever Misbehave – the other half of the observer effect: a probe that pays its own cost
- Bugs Written by AI Look Like Correct Code: An Epistemic Rewrite of Debugging – where this series’ epistemology starts
- Visual Studio Debug vs Release, Decoded (Chinese) – two physical systems on one machine
- WinDbg for User-Mode Debugging: A Field Guide (Chinese) – a systematic tutorial for Appendix B’s commands
- Every Stack Is Innocent: A Differential Diagnosis of the Hung Program – sentence two: the enemy is time, not the stack
- “It Works on My Machine”: A Complete Taxonomy of Environment Differences – why the same code is a different program on the other box
