You are debugging a production problem. You open the Dockerfile in the repository, read it, and reason about what the container does.
The container was not built from that file. It was built from a copy on the build host, edited during an incident eight months ago, and the two have been diverging ever since. Everything you concluded is about a file that has never run in production.
This post is about deployment provenance: knowing what your running artifact was actually built from, why the answer is so often "not what you think", and the cheap instrumentation that makes it knowable. For whoever maintains a deployment that grew rather than being designed.
How the copies diverge
Nobody decides to fork the build. It happens in three steps that are each individually reasonable.
Step one: an urgent fix. Production is broken at 23:00. The fix is one line in the Dockerfile or the compose file. Editing it on the build host and rebuilding takes two minutes; going through a pull request takes twenty. The right call is made under pressure, and the intention is to backport it tomorrow.
Step two: tomorrow does not happen. It is not written down anywhere, the incident is closed, and the divergence is now invisible because both files still exist and both look plausible.
Step three: it compounds. The next person makes the next urgent fix against the host copy, because that is the one that is live. Six months later the repository copy is a historical document that nobody has run.
The same happens to environment files, nginx configuration, systemd units, cron entries, and the deploy script itself. The pattern is always the same: a file that must change during an incident, in a location where changing it is faster than committing it.
Why it is worse than ordinary drift
Configuration drift on a server is a known problem with known answers. Build drift is worse for a specific reason: it makes the repository misleading rather than incomplete.
An absent file prompts a question. A present but stale file answers it, wrongly, with full confidence. Every subsequent decision inherits the error, and code review will happily approve a change to a file that has no effect on anything.
There are three concrete failures worth naming.
Reviews are theatre. A change to the repository Dockerfile passes review and never reaches production. Everybody believes it shipped.
Disaster recovery does not work. The rebuild-from-source plan reconstructs the version that was current before the drift began. You discover which parts were only on the host at the moment you no longer have the host.
Lockfile behaviour differs. If the host copy uses a plain install and the repository copy uses a frozen lockfile, or the reverse, the two builds resolve different dependency versions. Then the artifact in production contains packages that no lockfile in your repository describes, which is a supply chain problem as well as an operations one.
Finding out, in about ten minutes
Ask the artifact what it was built from, and compare.
# What the running container was built from, if anything labelled it
docker inspect <name> --format '{{json .Config.Labels}}' | jq .
# The image tag itself often carries a commit
docker inspect <name> --format '{{.Config.Image}}'
# Then compare the build inputs on the host against the repository
diff -u /path/in/repo/Dockerfile /path/on/buildhost/Dockerfile
If the labels are empty, that is the finding. An unlabelled image cannot tell you its own origin, which means every question about provenance becomes archaeology.
Then check the build host's checkout:
git -C /path/to/build/checkout log --oneline -1
git -C /path/to/build/checkout status --porcelain # uncommitted local edits
Output from that last command is the direct evidence: local modifications on a build host are the divergence, visible.
Making it knowable
Label every image at build time. One argument each, and it converts provenance from archaeology into a query.
docker build \
--label org.opencontainers.image.revision="$(git rev-parse HEAD)" \
--label org.opencontainers.image.source="$REPO_URL" \
--label org.opencontainers.image.created="$(date -u +%FT%TZ)" \
-t "service:$(git rev-parse --short HEAD)" .
Tag with the commit as well as with latest. A tag that names a commit means a glance at docker ps answers what is running, and it is what let us identify, during an incident today, that repeated deploys were rebuilding the same pre-fix commit.
Fail the build on a dirty checkout. If the build host has uncommitted changes, stop:
if [ -n "$(git status --porcelain)" ]; then
echo "refusing to build from a modified checkout" >&2
exit 1
fi
This is the control that prevents the whole class. It is also the one people disable first during an incident, so pair it with an escape hatch that is loud: an environment variable that permits it and records the fact in the image labels, so a build from a dirty tree is possible but never invisible.
Put the operational scripts in the repository. Drain scripts, deploy scripts, health probes. If it is load-bearing for a deploy, it belongs in version control regardless of whether it feels like application code. The test is simple: if this file were deleted right now, would anyone be able to reconstruct it?
Reconcile deliberately, once. Diff every build input between host and repository, commit the host version as the truth, then delete the host copy and point the build at the checkout. Doing this is usually an afternoon and it is the only way to stop paying for the divergence.
The concession
Editing on the host during an incident is often correct. The alternative, a pull request and a pipeline run while production is down, is not obviously better, and a policy forbidding it will be broken by whoever is on call at 03:00, correctly.
So the goal is not preventing the edit. It is making sure the edit cannot stay invisible: label the resulting image as built from a modified tree, and have the deploy leave a record that says so. Then the reconciliation is a task somebody can see rather than a discovery somebody makes later.
The implication
The question "what is running in production" should be answerable in one command, by anyone, without asking a person who was there.
Most systems cannot answer it, and the cost stays hidden until an incident, when every minute spent establishing what the artifact was built from is a minute not spent fixing the problem. The labels cost one line in the build. That is the whole intervention.
