Every programmer has said it. Every programmer has been burned by it.
Monday morning, a colleague pings you: “Pulled latest, the build’s broken, can you take a look?” You pull on your machine–clean, all green. You reply: “Works on my machine.” The chat goes quiet, and you can taste the eye-roll from across the office.
The phrase has a thoroughly ruined reputation. Someone made a badge out of it–“Works on My Machine™,” complete with “Certified” underneath, as if it were an official credential. It sits permanently on every “things programmers hate hearing” list, right next to “this requirement is simple, should be quick.” Say it in a bug thread and everyone hears: not my problem, you deal with it.
This article wants to mount a serious defense: that sentence is not an excuse. It’s testimony.
The speaker isn’t lying. The program really does work on their machine. And buried inside that sentence is one of the highest-value facts in all of debugging: this bug is environment-coupled. Its trigger isn’t in the code’s logic–or isn’t entirely. There’s a variable in the trigger equation hiding somewhere in the differences between two machines. Translated into debugging language: the differential experiment has already assembled itself. One machine is the good environment, the other is the bad one, and the list of differences between them is the suspect list.
A sentence treated as a punchline is actually the opening move of an investigation plan. The problem was never the sentence; it’s that in most teams, the investigation ends the moment it’s spoken. The correct plot is: the investigation starts there.
But to turn an excuse into testimony, you need a map: what kinds of environment differences exist, how each one commits its crimes, and what fingerprints they leave. This article draws that map.
I. The Mechanism: The Environment Is a Contract Nobody Wrote
To understand “why does behavior change when the machine changes,” you first have to demolish a quiet illusion: source code equals program. As if identical code must produce identical behavior.
Straighten out the chain of “how a program actually runs” and the illusion collapses. Your source is transformed by a compiler (or interpreter, or runtime) into an executable form; at runtime it asks the OS for memory, reads and writes files, fetches the time, sends packets; underneath it all stands the CPU’s instruction set and floating-point behavior; outward, it talks to databases, third-party services, networks. The source is just the first link in that chain.
As a formula:
Program behavior = f(source, input, environment)
Source is one of three independent variables. What makes “environment” the insidious one is that it was never declared. Functions have signatures, interfaces have docs, but no language offers a single line of syntax for “this program assumes the environment satisfies the following.” Yet every program silently carries hundreds of such assumptions:
- that the dependencies installed are the versions you tested with;
- that the filesystem agrees with you about case sensitivity;
- that the default encoding, timezone, and decimal separator are “the same everywhere”;
- that the CPU understands every instruction you used;
- that the network is reachable, the services are up, and the API still returns the old shape.
That unwritten list is what I’ll call the implicit environment contract. Every environment bug is, at bottom, a breach of this contract–by parties who have never seen the text: one side breaches unknowingly, the other suffers unknowingly.
This lens unifies a lot of “spooky” phenomena. The Debug/Release differences piece (Chinese) argued that the same source compiled debug and release is two different programs. That’s actually a special case of today’s topic–two environments on one machine. Zoom out: any two machines, any two environments, are two physical systems, each satisfying a different subset of the contract. Moving a program from environment A to B is like taking a contract that was only ever validated in A and trying to cash it in B.
So the full translation of “works on my machine” is: my environment happens to satisfy the unwritten contract; yours is in breach of some clause. Which clause–that’s exactly what the taxonomy in the next section is for.
II. The Taxonomy: Five Layers of Difference
Environment differences come in endless variety, but sort them by “how far from your code” and they collapse into five clean layers–five concentric rings, with your code at the center, each ring drifting in its own way:
| Layer | In one phrase | Why it drifts |
| 1 Dependencies | The things your code pulls in | Versions drift; the things you pull in pull in other things |
| 2 Toolchain | What turns source into a program | Every machine has its own compilers and runtimes |
| 3 System interfaces | The OS your program talks to at runtime | Same question, different systems, different answers |
| 4 Hardware | The machine your program stands on | Instruction sets, precision, and architectures have generations |
| 5 Outside world | Networks and data beyond your machine | Nothing you connect to is under your control |
For each layer, three things: the mechanism (why “different” breaks things), real cases, and the fingerprint (which symptoms should make you suspect this layer first). The fingerprint is the triage desk–worth committing to memory even more than the mechanism.
Layer 1: Dependencies–The Things You Pull In Are Drifting
Mechanism. No modern program is written from scratch. Your code stands on hundreds of packages, and those packages stand on their own dependencies–what you think of as a dependency tree is actually a dependency web. The problem: the version you declare is usually not a point but a range. ^1.2.3 means “anything from 1.2.3 up to, but not including, 2.0.” Today that installs 1.2.3; six months from now, 1.9.0–same declaration, different contents. If the lock file never made it into version control, or someone ran an “install latest” command, the two machines aren’t holding the same web.
Case. This layer has an industry-grade monument: left-pad. In March 2016, a developer–embroiled in a trademark dispute–unpublished all of his npm packages in one go. One of them was left-pad, a function that pads strings on the left. Eleven lines. Not in your package.json? Doesn’t matter: it was in your dependency’s dependency’s dependency. That day, builds of Babel, React, and a raft of flagship projects failed across the world. Eleven lines of code set the market price of “I assumed my dependencies were stable.”
The everyday version is quieter: a new hire joins, clones the repo, runs npm install, and the service dies on startup–while your two-year-old nodemodules sits there like a rock. The code is byte-for-byte identical. What differs is the version combination of those hundreds of packages inside nodemodules. npm eventually added npm ci–install strictly from the lock file, structurally incapable of producing a second outcome. A dedicated command had to be invented. That’s the official confession that this layer of problem exists.
Fingerprint: Reproduces only on a fresh clone or a fresh dependency install. Every old machine works. When you see that pattern, diff the dependency trees first. Don’t touch the code.
Layer 2: Toolchain–What Turns Source into a Program Is Different
Mechanism. The same source, handed to different compiler versions, yields two flatly different programs. And the difference goes beyond a version number: new compilers fix bugs, change optimization strategies, swap standard library implementations; every runtime release adds and removes APIs. C and C++ keep an extra ledger–undefined behavior (UB). Which means different compiler versions can hand down different verdicts on the same UB: gcc 12 acquits (“happens to work”), gcc 14 sentences it to death. Not one character of code changed; the compiler aged one generation; the program’s fate flipped.
Case. This layer’s most famous error message is burned into everyone who has ever deployed on Linux:
version `GLIBC_2.34' not foundYou compiled on Ubuntu 22.04; the binary references symbols from the new glibc. You carry it to Ubuntu 20.04, and the dynamic linker goes on strike on the spot. The code didn’t change–what changed is the machine that compiled it. The gentler variants are worse: Python 3.12 removed distutils, and a pile of build scripts quietly died; type gcc on macOS and you’re actually running clang–same command, different thing; whether your Visual Studio has the v142 or v143 MSVC toolset installed can give the same C++ source subtly different behavior. Not a crash–just “the numbers are slightly off from before,” which is more agonizing than a crash.
Fingerprint: Rebuilding changes the behavior; copying the built artifact over unchanged does not. This fingerprint cleanly separates Layer 2 from all the others. Memorize it–the two experiments in Section III are designed around it.
Layer 3: System Interfaces–Same Question, Different Answers
This is the disaster zone, with the most sub-species. The shared mechanism in one sentence: your program asks the operating system a question, and different systems give different answers to the same question. The program isn’t wrong; the question isn’t wrong. What’s wrong is the assumption that the answer is unique.
Paths and filesystems. NTFS is case-insensitive by default, ext4 is strictly case-sensitive, APFS (macOS default) is insensitive again. So #include "String.h" compiles on Windows–the filesystem found string.h for you–and dies on a Linux CI with “no such header.” The classic GitHub trap has the same DNA: two files in a repo differing only in case coexist happily on a Mac and collide on checkout on Linux. Windows adds a few local specialties: the default path-length cap of 260 (MAX_PATH)–nest your repo a little deeper and builds fail mysteriously; the space in C:\Program Files, which has ambushed every generation of scripts; reserved names like CON, AUX, and NUL–you thought you picked a filename, but it’s a system device.
Encoding and line endings. The collective trauma zone–doubly so if you’ve worked with Chinese Windows. A Chinese Windows system defaults to code page 936 (GBK); the Linux world defaults to UTF-8. A UTF-8 source file without a BOM, read as GBK by MSVC, turns comments into garbage or fails the build outright. Console output is the same movie: the program prints UTF-8, the console renders GBK–enter the legendary mojibake 锟斤拷. (Western Windows has its own dialect: smart quotes and dashes in Windows-1252 turning “don’t” into “don’t.”) A quick hello to an old friend while we’re here: 烫烫烫–roughly “hot hot hot.” MSVC debug builds fill uninitialized stack memory with 0xCC, and two 0xCC bytes decoded in GBK spell exactly that character (its origin story is in the Debug/Release differences piece (Chinese)). Line endings, meanwhile, are the eternal cross-platform commuter: a shell script saved on Windows and run on Linux reports /bin/bash^M: bad interpreter. That ^M is a carriage return–one invisible character, one ruined afternoon.
Time and timezones. When your program asks the system “what time is it,” the answer depends on whether it’s local time or UTC, on the version of the timezone database (tzdata), on the daylight-saving rules in force. tzdata is “factual data” that gets updated: Egypt reinstated DST in 2023, and every machine with a stale tzdata shifted all its appointments by an hour after April–nobody changed any code; the rules themselves changed. In any system deployed across timezones, the offhand decision “store local time or UTC” will eventually come to collect rent, in the currency of bugs.
Locale. In German, the decimal separator is a comma: 1,5 is the native spelling. A program that parses numbers according to the system locale reads “1.5” as 15 on German Windows. Database collations run the same con: utf8mb4generalci is case-insensitive; switch collations and the same SQL query changes its comparison behavior and result ordering.
Permissions and resource limits. A program that ran fine under your admin account, run as a standard user: a legacy app without a compatibility manifest writes to Program Files and gets silently redirected by UAC into the VirtualStore–the program believes the write succeeded; you open the folder, and the file simply isn’t there. The Linux dialect of the same story: Too many open files (ulimit defaults to 1024). One story, two accents.
Fingerprint: Only appears on a particular OS, language/regional setting, or account. The direction of investigation is to put the two machines’ system settings side by side–not to stare at the code.
Layer 4: Hardware–The Floor Beneath Your Feet Is Different
Mechanism. The instructions your compiler emits are ultimately executed by a CPU, and CPUs have generations: your dev box supports AVX2; that old server in the customer’s rack does not. Floating-point behavior drifts with hardware and compiler policy too: x87 used 80-bit intermediate precision internally, SSE is 64-bit from the ground up–the same line of code can produce a different last digit on two generations of hardware. Irrelevant to most business logic; a breeding ground for flakiness in numerically sensitive algorithms and tests that assert exact values.
Case. The -march=native (“optimize for this CPU”) you tossed in during performance tuning is this layer’s homemade landmine: the binary flies on your machine, and on a machine lacking the new instruction set it doesn’t slow down–it crashes on the spot with SIGILL (illegal instruction). The more frequent plotline in recent years is architecture differences: running an x86 image on an Apple Silicon Mac is either absurdly slow (QEMU translating instruction by instruction) or segfaulting outright. When a colleague says “it works in my Docker,” remember to ask one more question: on which architecture?
Fingerprint: Only appears on a different physical machine, or a different VM generation. On the same machine, nothing you do reproduces it.
Layer 5: The Outside World–Everything Past the Machine’s Edge
Mechanism. The first four layers are at least inside the chassis. Layer 5 lives past the edge: the network your program connects to, the services it calls, the data it reads–none of it answers to you. This layer also loves to conspire with time–“the environment hasn’t changed” is usually an illusion; it just changed slowly and quietly, beneath your notice.
Case. The number-one repeat offender is the hosts file: three months ago, for an integration test, you pointed api.example.com at 127.0.0.1 to serve a mock, and forgot to remove it. Ever since, everything works on your machine–because you have never once talked to the real backend. The HTTP_PROXY environment variable is its accomplice. Next comes latency: a 1ms local call lets you synchronously call it a hundred times inside a loop, no sweat; in production, one 300ms hop across AZs avalanches into timeouts. Finally, data: the dev database holds 100 rows of clean test data; production’s ten million rows contain empty strings, oversized fields, and mojibake entered a decade ago. Your query glides through the dev environment and ends up on a slow-query report in production.
Fingerprint: Behavior changes after you cut the network, switch data sources, or switch accounts. The converse also holds: an environment where every mock is green is precisely the one most worth suspecting.
Five layers surveyed. Compress them into one triage table–it deserves a spot on your wall:
| What you’re seeing | Suspect first |
| Reproduces only after fresh clone / fresh install | Layer 1: Dependencies |
| Reproduces only after rebuild; the copied artifact is fine | Layer 2: Toolchain |
| Only on a specific OS / locale / account | Layer 3: System interfaces |
| Only on a different physical machine or VM generation | Layer 4: Hardware |
| Behavior changes when network or data source changes | Layer 5: Outside world |
One caution: real bugs can operate across layers (dependency drift plus a newer compiler, jointly escorting a latent UB to the gallows). The table gives you an opening move, not a final verdict.
III. The Method: Differential Testing and Bisection
The map exists. How do you move at the scene of the crime? Three steps.
Step 1: Two experiments that cut the suspect pool in half
The entire discipline of debugging environment problems reduces to one rule: change exactly one variable at a time. Concretely, one experiment in each direction.
Experiment A: Freeze the artifact, swap the environment. Copy the built artifact–binary, package, image–unchanged onto the failing machine and run it. Still crashes: the problem is on the environment side (Layers 3/4/5) or in the artifact-environment interaction. Doesn’t crash: reverse course and investigate how it’s run–launch arguments, working directory, account, environment variables.
Experiment B: Freeze the environment, rebuild. Rebuild from source on the failing machine (or in a clean container) and run that. Behavior changed: Layer 2 (toolchain) is the prime suspect. Behavior unchanged: the toolchain is acquitted.
Cross A with B and the suspect pool collapses by half or more. The most common way people botch this step is one sentence: “I pulled the code and rebuilt it on his machine–still crashes.” That operation changed two variables at once. Artifact and environment are now blended together, and the experiment counted for nothing.
Step 2: The environment fingerprint–let the machine confess
Once the scope has narrowed, the question becomes: what exactly differs between the two machines?
You will not get this answer by asking. “What version of Node do you have?” “Pretty recent, I think.” “What’s your system encoding?” “Default, probably?” Human memory of one’s own machine is unreliable enough to belong in a psychology textbook. The correct approach is to make the machine confess: one collection script, run on both sides, output redirected to a file, then diff the two files. I call it the environment fingerprint. It doesn’t need to be perfect; it only needs to capture the few dozen items that cause the most trouble:
# env-fingerprint.ps1 (Windows)
$lines = @()
$lines += "=== OS ==="
$lines += (Get-CimInstance Win32_OperatingSystem |
Select-Object Caption, Version, BuildNumber | Format-List | Out-String)
$lines += "=== CPU ==="
$lines += (Get-CimInstance Win32_Processor |
Select-Object Name, AddressWidth | Format-List | Out-String)
$lines += "=== Locale / Codepage / TZ ==="
$lines += "Culture=$((Get-Culture).Name) TZ=$(tzutil /g)"
$lines += (chcp | Out-String)
$lines += "=== Toolchain ==="
foreach ($t in "git","node","python","dotnet","cmake") {
$c = Get-Command $t -ErrorAction SilentlyContinue
if ($c) { $lines += "$t => $($c.Source)"
$lines += ((& $t --version 2>&1 | Select-Object -First 1) | Out-String) }
else { $lines += "$t => (missing)" }
}
$lines += "=== Lockfiles ==="
foreach ($f in "package-lock.json","yarn.lock","Cargo.lock","go.sum") {
if (Test-Path $f) {
$lines += "$f => $((Get-FileHash $f -Algorithm SHA256).Hash.Substring(0,16))"
}
}
$lines += "=== Env Vars ==="
$lines += (Get-ChildItem env: | Sort-Object Name |
Format-Table Name, Value -AutoSize | Out-String)
$outFile = "fingerprint-$env:COMPUTERNAME.txt"
$lines | Out-File $outFile
Write-Host "done: $outFile"#!/usr/bin/env bash
# env-fingerprint.sh (Linux / macOS)
out="fingerprint-$(hostname).txt"
{
echo "=== OS ==="
uname -a
sw_vers 2>/dev/null
echo "=== glibc ==="
ldd --version 2>/dev/null | head -1
echo "=== CPU ==="
grep -m1 "model name" /proc/cpuinfo 2>/dev/null
grep -o -m1 -E "avx2|avx512f" /proc/cpuinfo 2>/dev/null | sort -u
echo "=== Locale / TZ ==="
locale
date "+%Z %z"
echo "=== Toolchain ==="
for t in git gcc clang node python3 cmake; do
if command -v "$t" >/dev/null 2>&1; then
echo "$t => $(command -v $t) ($($t --version 2>&1 | head -1))"
else
echo "$t => (missing)"
fi
done
echo "=== Lockfiles ==="
for f in package-lock.json yarn.lock Cargo.lock go.sum; do
[ -f "$f" ] && echo "$f => $(sha256sum "$f" | cut -c1-16)"
done
echo "=== Env Vars ==="
env | sort
} > "$out"
echo "done: $out"Run it once on each side, then:
diff fingerprint-MACHINE-A.txt fingerprint-MACHINE-B.txtThe philosophy of this approach fits in one sentence: don’t ask what people think is installed; read what the machine confesses. Version numbers, paths, locale, timezone, environment variables, lock file hashes–every line comes straight from the machine, without a single round trip through human memory.
(Extend the scripts to match your stack: write Rust, add rustc/cargo; write Java, add java -version. What matters isn’t the exact capture list–it’s the discipline of running the same script on both sides. The two halves of a diff must be measured with the same ruler.)
Step 3: Treat the diff as a lineup, interrogate one suspect at a time
Every line of the diff is a suspect. Bisection again: pick one difference, neutralize it across both machines (align the version, set the locale the same, sync the environment variable), run once. The behavior changes–the culprit is in custody. The behavior doesn’t change–acquitted and released, next suspect. If you’ve neutralized everything and the case is still open, go back and re-read the case file: either the difference isn’t in your capture list (something the script didn’t cover), or this was never a spatial difference but a temporal one. “It worked last week” doesn’t mean two machines differ; it means this week differs from last week: an expired certificate, a rotated token, a third-party API redesign. Layer 5, the hidden flavor.
One final handoff: if the failing machine is somewhere you can’t reach–a customer environment, a production cluster–then even “run a script” is a luxury. Stop dreaming of interactive debugging. Have the field capture a dump file and send it back for post-mortem analysis.
IV. Prevention: Turn Snowflakes into Cattle
Investigation is damage control–stopping the bleeding. Once it’s stopped, let’s talk about getting cut less often.
The prevention philosophy in one sentence: turn environments from unreplicable one-offs into mass-producible artifacts. Operations has a classic pair of metaphors–snowflakes and cattle. A snowflake server is unique, hand-tuned, impossible to rebuild when broken–you can only repair it. Cattle servers come in herds, perfectly interchangeable; one dies, you lead in another, and nobody grieves. Your dev environment, build environment, deployment environment–each deserves one question: snowflake or cattle?
Prescriptions, layer by layer:
- Layer 1 (dependencies): lock files into version control (package-lock.json, Cargo.lock, go.sum); strict installs in CI (
npm ci); floating ranges evicted from package.json. Turn “installing dependencies” from everyone-for-themselves into shipping against a manifest. - Layer 2 (toolchain): pin toolchain versions (.nvmrc, rust-toolchain.toml, the version in setup-python), or just build in a container. Go further: artifacts get built, stored, and distributed by CI alone–“build once, run everywhere”–and the phrase “built on my machine” gets fired from the process.
- Layer 3 (system interfaces): devcontainers and Dockerfiles freeze system configuration into version control; at the code level, stop borrowing–standardize on UTF-8, on UTC storage, on locale-independent parsing APIs.
- Layer 4 (hardware):
-march=nativeis banned from release builds; add multi-architecture builds to CI (x86-64 plus arm64) so architecture differences surface before production. - Layer 5 (outside world): hosts overrides get registered and get expiry dates; mocks stay out of personal configuration; every outbound call gets an explicit timeout; test data comes pre-seeded with dirty data.
More valuable than any specific measure: two disciplines.
Burn it down, periodically. The textbook symptom of a snowflake environment is “we don’t dare reinstall it”–nobody can say what undocumented configuration still lives in there. The test for snowflake versus cattle is a single question: delete it, rebuild from what’s in version control alone–how long until it’s whole again? If you can, the environment is an artifact. If you can’t, it’s an archaeological relic. Quarterly is a good burn schedule. Whatever survives the fire is real configuration.
CI is the reference machine. Local environments are one per person, each drifting its own way. The only party that doesn’t take sides is CI. It belongs to no one and honors only what’s in the repository–which makes it the only place the implicit environment contract is ever written down. A green CI run is, approximately, the contract being honored in full. If it doesn’t run on your machine, your machine lines up with CI–not the other way around.
And honestly: prevention has a limit
This last subsection may be the most important sentence in the article: everything above can only pin the dependencies you are aware of.
A lock file locks what package.json names; it cannot lock “you unknowingly depend on some behavior of glibc.” Docker pins the base image tag; it cannot pin the host kernel, the GPU driver, or that one machine’s particular luck. Container parity guarantees “my side equals your side” only within the boundary you drew–the contract clauses you never knew existed stay outside the list, always.
This isn’t pessimism; it’s precision: environment differences can be compressed, never eliminated. The engineering win is not eliminating them, but crushing “difference-induced incidents” from daily routine down to rare probability–and, when the rare thing happens, holding a five-layer triage table and a fingerprint script that close the case in hours instead of weeks. One line: prevention makes the phrase rare; method makes the phrase harmless.
V. Closing: Turn the Excuse into a Starting Point
Back to the sentence they made a certification badge out of.
Its reputation is ruined not because it’s wrong, but because it’s almost always spent where the plot ends: the sentence is spoken, both sides shrug, everyone goes home, and the bug continues living on the failing machine. But by this article’s argument, it belongs where the plot begins–
- It’s testimony: proof that the bug depends on the environment, that an environment variable hides in the trigger equation.
- It’s half an experiment: the good environment and the bad one are already in place; all that’s left is to pull up the list of differences and diff them.
- It’s an invitation: the difference between two machines is the suspect list.
So the next time you hear “works on my machine,” instead of rolling your eyes, try answering with the line that stops the room:
“Great–then let’s diff your environment against mine.”
On a team that can answer that way, the mockery on the badge stops working. Because once “works on my machine” stops being a disclaimer and becomes an experiment design, environment differences stop being mysticism and become engineering.
Related Posts
- The Harder You Look, the Less It’s There: Heisenbugs and the Observer Effect in Debugging–observation changes the system; this piece covers the other half: even unobserved, systems differ by environment
- Bugs Written by AI Look Like Correct Code: An Epistemic Rewrite of Debugging–when code is no longer typed by humans line by line, breaches of the implicit environment contract only multiply
- Debug vs. Release in Visual Studio, In Depth (Chinese)–the mechanistic special case of “the same source is two programs”
Leave a Reply