Three automated agents want to deploy the same service. There is one lock. Each agent takes it, does its work, releases it, and retries when it cannot get it.
Nothing here is broken. Every agent is correct in isolation, the lock does its job, and the deployments serialise exactly as designed. The service still ends up restarting every few minutes for an hour, because nobody wrote the rule that says how often an agent may ask.
As more of the work around a deployment pipeline gets handed to automated agents, this stops being a curiosity and becomes an operating problem. This post is about what goes wrong when several autonomous actors share one piece of infrastructure, and the small set of conventions that fix it.
What contention looks like when the actors are automated
A human who cannot get the deploy lock goes and does something else, and tries again in twenty minutes. Their retry behaviour is shaped by boredom, which turns out to be an excellent backoff algorithm.
An agent retries at whatever interval it was told, and the default is usually far too fast. The result is a log full of a single line:
[21:53:52] Another prod deploy run is active; exiting.
[21:53:53] Another prod deploy run is active; exiting.
[21:53:54] Another prod deploy run is active; exiting.
A hundred of those a minute. None of them is an error. Nothing alerts. The lock is working perfectly, and the only visible symptom is that the log is now useless for the incident you are actually trying to diagnose, because the real events are buried under polling.
The second symptom is worse. Each successful acquisition performs a real deployment, which is a real container swap, which is a real interruption of whatever the service was doing. Three agents each deploying every few minutes produce a service that restarts continuously, and every restart is legitimate. There is no bug to find.
The four rules that resolve it
1. Back off, and back off a lot. A retry interval measured in seconds is polling. For something as expensive as a deployment, the floor should be tens of seconds and the backoff should grow. An agent that has failed to get the lock ten times should be waiting minutes, not still asking every second.
2. Check whether the work is still needed before doing it. This is the rule that removes most of the waste. An agent waiting to deploy commit X should, on acquiring the lock, check whether the running artifact is already commit X. Very often another agent deployed it while this one was queued, and the correct action is to exit satisfied rather than to deploy again.
running=$(docker inspect svc --format '{{.Config.Image}}')
case "$running" in
*"$TARGET_COMMIT"*) echo "already deployed, nothing to do"; exit 0 ;;
esac
Verify by artifact, not by exit code. An agent that concludes "my deploy failed, retry" because a previous run exited non-zero will loop forever if the deploy actually succeeded.
3. Say who you are. A lock that records only that it is held is much less useful than one that records who holds it and why:
echo "$AGENT_NAME pid=$$ since=$(date -Iseconds) reason=deploy-$SERVICE" > "$LOCKFILE"
The cost is one line. The benefit is that the next agent, or the next human, can see whether the holder is a legitimate long build or a process that died holding the lock, and can say so to a person instead of retrying blindly.
4. Announce long operations where the others can see. If agents share an operational channel, a message on acquisition and on release converts invisible contention into something observable. This is the automated equivalent of saying "I'm deploying" in a room.
Detecting a stale lock without breaking things
The failure that tempts everyone into deleting lock files: a process dies holding the lock and every other agent waits forever.
Do not delete a lock file because it has been held a while. A long build looks identical to a dead holder from the outside, and removing a lock from a live deployment produces two concurrent deployments, which is the thing the lock existed to prevent.
Use a lock that releases itself. flock on a file descriptor is released by the kernel when the process exits, whatever way it exits, so a dead holder frees the lock automatically:
exec 9>"$LOCKFILE"
flock -n 9 || { echo "held by: $(cat $LOCKFILE.meta 2>/dev/null)"; exit 75; }
If you must use an advisory lock file, put the holder's pid in it and let a waiter check whether that pid is alive before concluding anything. And if a waiter does decide a lock is stale, the right action is to tell a human, not to break it.
The thing that is genuinely new
What is different about agent contention is not the locking, which is an old problem with good solutions. It is that the actors can be numerous, fast, and unaware of each other, and that each one is individually well-behaved.
Nothing in the traditional toolkit alerts on this. There is no error to catch, no exception to log, no threshold crossed. The system is doing exactly what everyone asked it to do, and the aggregate behaviour is harmful.
So the monitoring has to be aggregate too. Two metrics catch it:
- Lock acquisition attempts per minute. A sudden rise means polling, whatever the individual agents think they are doing.
- Deployments per hour per service. If this exceeds what any human intended, something is deploying in a loop regardless of how correct each iteration is.
Alert on both. They are cheap, and they are the only signals that see the pattern.
The concession
Coordination has a cost, and for two agents that rarely collide it is not worth building. A short retry loop on a rarely-contended lock is fine, and adding a distributed coordination layer to solve a problem you do not have is its own mistake.
The threshold is roughly when the number of automated actors exceeds the number of people who know they exist. Past that point, the assumption every agent makes, that it is the only one acting, stops being true, and the conventions above are what replace the shared awareness a small team had for free.
The implication
The interesting failure is not that a lock was contended. It is that every participant behaved correctly and the outcome was still bad, which means code review of any single agent would have found nothing.
As more operational work moves to autonomous actors, that shape will become more common: correct components, harmful aggregate, no error anywhere. The defence is to make the actors aware of each other, and to monitor the aggregate rather than the parts.
