Your deployment script stops the old container and starts the new one. Everything in between was doing work, and that work is gone.
For a stateless request handler this is a solved problem: the load balancer stops sending new requests, in-flight ones finish within a few seconds, the container exits. For anything that runs longer than a request, a scan, an import, a report, a video encode, a batch job, the same mechanism silently destroys work that had been running for twenty minutes, and the user sees a job that never finished with no error attached to it.
This post is how to drain instead of kill, for whoever owns a deployment that also runs long jobs.
What actually happens at container stop
docker stop sends SIGTERM, waits ten seconds, then sends SIGKILL. Kubernetes does the same with a default terminationGracePeriodSeconds of thirty.
Both numbers are wrong for long work, and the second one is the dangerous one. SIGKILL cannot be caught, so no cleanup runs, no status is written, and the job row stays in whatever state it was in. Usually that is RUNNING, forever, which is worse than FAILED because nothing retries it and nothing alerts on it.
Three things follow. Your process must handle SIGTERM. The grace period must be longer than your longest job, or you need a way to not have long jobs running when you deploy. And any job left behind needs to be recoverable, because eventually one will be.
The three strategies
Wait for quiet. Before stopping anything, ask the service whether it is busy. Poll until it says no, then deploy.
# Ask the running service whether work is in flight.
for i in $(seq 1 120); do
active=$(curl -sf http://127.0.0.1:8080/internal/drain-status | jq -r '.active')
[ "$active" = "0" ] && break
echo "waiting: $active job(s) in flight"
sleep 10
done
Simple, and it needs the service to expose a truthful count. The weakness is a service that is never quiet, where this waits forever. Always cap the wait and decide explicitly what happens at the cap.
Stop accepting, finish what you have. On SIGTERM, refuse new work, let current work finish, then exit. This is the correct behaviour and it composes with a long grace period.
@PreDestroy
void drain() {
accepting.set(false); // new jobs rejected from here
executor.shutdown(); // queued work still runs
if (!executor.awaitTermination(30, TimeUnit.MINUTES)) {
log.warn("drain timed out, {} job(s) abandoned", executor.shutdownNow().size());
}
}
Set the platform's grace period above that timeout or the platform kills you mid-drain, which is the same outage with extra steps and a misleading log line.
Make the work resumable. The only strategy that survives a power failure, and the only one that scales past a single deploy. Checkpoint progress, and on startup find jobs marked running with no live owner and either resume or fail them explicitly.
The lease pattern is the usual implementation: a worker claims a job with an expiry, renews the lease while working, and a sweeper reclaims anything whose lease has expired. It costs a column and a scheduled task, and it turns a killed job from a permanent mystery into a delay.
Where this goes wrong in practice
The drain script lives on the server and not in the repository. Extremely common for operational scripts, because they get written during an incident. Then it is not reviewed, not tested, and absent from the disaster recovery you would need it for. If a script is load-bearing for a deploy, it belongs in version control, whatever its file extension.
The health check flips before the drain finishes. If your readiness probe reports unhealthy the moment SIGTERM arrives, but the process then takes ten minutes to drain, monitoring shows a down service for ten minutes and somebody will restart it. Report draining as a distinct state, and make sure the dashboard knows about it.
Only one path is drained. The HTTP server drains cleanly while the message queue consumer, the scheduler, and the hand-started worker container nobody remembers are all killed outright. A drain covers the deployment unit, and anything outside it is not covered. Enumerate the processes that do work, not the ones the deploy script happens to name.
The timeout is shorter than the work. A thirty-minute grace period and a forty-minute job is a policy that abandons the tail of your workload. Measure the actual distribution before picking the number, and pick against the 99th percentile rather than the median.
What to tell the user
Even a perfect drain abandons something eventually. That case deserves a real outcome, not silence.
Mark the job with a distinct state, INTERRUPTED rather than FAILED, because it says something different: nothing was wrong with the work, the platform stopped it. Make it retryable with one action. If it was triggered by a person, tell them. A job that dies without a message trains users not to trust long operations, and that distrust is much harder to fix than the drain.
The concession
There is a legitimate argument that all of this is the wrong shape, and that long work should not live in the deployable unit at all. Put it in a queue, run workers separately, deploy them on their own cadence, and the problem mostly dissolves.
That is right, and it is a larger change than adding a signal handler. The intermediate position is worth naming: drain what you have now, expose a truthful busy count, then move the long work out when you next have the room. Teams that wait for the correct architecture usually ship neither.
The implication
Every deployment is an interruption. The question is only whether the interruption is handled or discovered.
Handling it costs a signal handler, a status endpoint, a grace period that matches your workload, and a state for work that did not survive. That is an afternoon. Discovering it costs a user telling you their scan has been running since Tuesday.
