A long-running job has been going for forty minutes. Is it working, or is it stuck?
Most systems cannot answer. The job writes nothing until it finishes, so the only signals available are indirect: CPU usage, open connections, whether the process still exists. Every one of those lies in a way that matters, and this post is about why, and what to emit instead.
Written for anyone operating work measured in minutes rather than milliseconds: scans, imports, encodes, migrations, training runs, batch reports.
The indirect signals and how each one lies
CPU usage. High CPU looks like progress and often is not: a retry loop, a regex backtracking, a spin on a lock. Low CPU looks like a hang and usually is not, because most long jobs are dominated by waiting on the network or disk. A job at 3 percent CPU may be perfectly healthy and a job at 100 percent may be going nowhere.
Connection counts. Tempting for anything that talks to the network, and it counts sockets rather than work. Connection pools hold idle connections open, so the count stays flat whether requests are flowing or not. Keep-alive makes it worse. A rising count often means connections are failing to be reused, which is a symptom of trouble rather than evidence of progress.
Process liveness. Answers whether it crashed, which is a different question. A deadlocked process is alive.
Log volume. Better than the others, and still noisy: a job that logs per error goes quiet when things are going well, so silence is ambiguous in the worst possible way.
The common defect is that all four are proxies. They correlate with progress in the normal case and decouple in exactly the abnormal case you are trying to detect.
Emit progress as a first-class thing
The fix is not clever. It is to make the job say what it is doing, in units that mean something to the person asking.
A monotonic counter of completed units. Not a percentage. Percentages require knowing the total, which for most real work you do not, and a percentage that sticks at 80 tells you nothing about whether 80 was reached a second ago or an hour ago.
A timestamp of the last completed unit. This is the single most useful field and it is the one usually missing. "Last unit completed 4 seconds ago" and "last unit completed 38 minutes ago" are completely different situations that every indirect signal renders identically.
The current phase. Long jobs have stages: discovery, fetch, analysis, report. Knowing which one it is in turns "stuck" into "stuck in fetch", which is most of the diagnosis.
A stable job identifier, so all of this is queryable per job rather than per process.
{
"job_id": "scan_8812",
"phase": "analysis",
"units_done": 1470,
"units_total": null,
"last_unit_at": "2026-09-17T21:04:12Z",
"started_at": "2026-09-17T20:31:55Z"
}
units_total is deliberately nullable. Emitting null is honest; emitting a guess produces a progress bar that goes backwards, which destroys trust in the whole display.
Where to put it
An endpoint the job serves, if it is a service. GET /jobs/{id}/progress. Cheapest to build, and it works for an operator with curl.
A row the job updates. Better for batch work, because it survives the process dying: if the job is killed, the row holds the last known position, which is exactly what you want during the investigation.
A structured log line per N units, with the fields above. Works everywhere, needs no infrastructure, and your log aggregator turns it into a timeline for free. The trick is to log every N units rather than every unit, so a large job does not drown the logger.
Emit on a schedule as well as per unit. A job whose unit takes twenty minutes should still emit a heartbeat every thirty seconds saying it is alive and in which phase, or it is indistinguishable from stopped for twenty minutes at a time.
The alert that works
Once last_unit_at exists, the alert is obvious and it is the one that catches real stalls:
Alert when a job is in a running state and
now - last_unit_atexceeds the 99th percentile unit duration by some margin.
That fires for the case that matters, a job that has stopped making progress while still claiming to run, and it does not fire for a job that is simply large. No proxy metric can express that condition, which is the whole argument for emitting it directly.
The second alert worth having: a job in a running state whose process no longer exists. This is the one that catches the killed-mid-flight case, where the row says RUNNING forever because nothing was around to update it.
Make it visible to whoever is waiting
If a person triggered the job, they need the same information, and they need it in their units rather than yours. "Analysed 1,470 of your files" is useful. "Phase 3 of 7" is not, because they do not know what the phases are.
And when it finishes, distinguish the outcomes. Completed, completed with errors, and did not finish are three different results, and collapsing the third into either of the others is how a killed security scan gets recorded as a clean one.
The concession
Instrumenting progress has a real cost, and it is not the code. It is that once you emit a number, people watch it, and they will ask why it is slower than last week. Some of those questions are valuable and some are noise, and you have volunteered for all of them.
There is also a genuine case for not doing this: a job that reliably takes under a few seconds does not need progress, it needs to either work or fail. The threshold is roughly whether a human will ever sit and wonder. Below that, skip it.
The implication
The question "is it working" is asked during an incident, under time pressure, by somebody who did not write the job. Every indirect signal available to them is a proxy that decouples from reality precisely when it is being consulted.
One timestamp fixes it. If you add nothing else, add the time the last unit of work completed, and expose it somewhere a person can read without attaching a debugger.
