Your CI Job Is Not Hung, It Is Slower Than Your Timeout

A test suite that takes 700 seconds, run under a harness whose default timeout is 600, does not fail as "too slow". It fails as killed, usually with no output, and every instinct you have says the tests are hung.

So you go looking for a deadlock. There is no deadlock. This post is about telling "slower than the limit" apart from "stopped making progress", why almost every tool defaults to conflating them, and the small changes that stop it costing an afternoon.

Why the two look identical

A timeout kills a process. That is all it does. Whatever the process was doing is gone, along with anything it had buffered and not flushed.

Three properties make the result indistinguishable from a hang:

Output is buffered. Test runners, language runtimes, and CI log collectors all buffer. A process killed mid-run loses whatever had not been flushed, so the log ends partway through, at an arbitrary point, exactly as it would if the process had stopped there.

The exit signal is the same. SIGKILL from a timeout and SIGKILL from an orchestrator evicting your job look identical from outside. The exit code is 137 either way, and it tells you nothing about the cause.

The error message describes the limit, not the reason. panic: test timed out after 10m0s is a true statement that does not distinguish a genuine deadlock from a suite that legitimately needs eleven minutes.

The defaults that catch people

Worth knowing rather than discovering:

• Go: go test has a 10 minute default per test binary. Not per test, per package binary. A package whose tests total 11 minutes fails without any individual test being slow. Pass -timeout 25m.
• Jest: 5 seconds per test by default. Generous for unit tests, hopeless for anything touching a container.
• pytest: no timeout by default, which is its own problem, since a genuine hang runs until the CI job limit.
• GitHub Actions: 360 minutes per job, 6 hours, but the step-level timeout-minutes you set for tidiness is usually the one that fires.
• Docker healthcheck: covered by its own start period, which is why a slow-starting container and a crash-looping one look the same in docker ps.

The pattern across all of them: the default was chosen for the common case, and your slowest package is not the common case.

Distinguishing them in practice

The question is whether the process was still making progress when it died. So make progress observable.

Print something per unit of work, unbuffered. If the log shows test 412 of 900 at the moment of death, and the previous line was two seconds earlier, the suite was running fine and simply ran out of clock. If the last line is forty minutes old, that is a hang and now you have the name of the test that hung.

go test -v -timeout 25m ./... 2>&1 | ts        # timestamp every line
pytest -v --timeout=60 --timeout-method=thread # per-test limit, not per-suite

Per-test timeouts are the stronger tool. A per-suite limit tells you the total was too large. A per-test limit names the test, and a hang is nearly always one test rather than all of them.

Get a stack dump instead of a kill. This is the part people skip and it is the one that answers the question outright. Several runtimes will dump every goroutine or thread on timeout, which either shows you a thread blocked on a lock, or shows you threads doing ordinary work.

kill -QUIT <pid>     # JVM: full thread dump to stdout
kill -ABRT <pid>     # Go: goroutine dump before dying
py-spy dump --pid <pid>   # Python, no restart needed

A blocked thread waiting on a mutex is a hang. Threads moving through application frames mean it was working, and your limit is wrong.

Measure before you raise. Run the thing locally with no timeout and time it. If it takes 700 seconds, the answer is a 25 minute limit and a note explaining why, not a 12 minute limit that fails again next month when the suite grows.

Do not silently raise every limit

The reflex, once bitten, is to set every timeout to an hour. That trades a false positive for a worse false negative: a genuine deadlock now burns an hour of CI and holds a runner, and the feedback loop that would have caught it is gone.

The position that holds up:

• Per-test timeouts tight, sized to the slowest legitimate test plus generous margin. These catch real hangs and name them.
• Per-suite timeouts loose, sized from measurement, as a backstop rather than a control.
• Job-level timeouts loose but present, because a runner held forever is a capacity problem for everyone else.
• Comment every non-default value with the measurement that produced it. A bare timeout: 25m invites someone to tidy it back down.

Where this gets expensive

The cost is rarely the timeout itself. It is the misdiagnosis that follows.

Someone concludes the suite is flaky and adds a retry. Now a slow suite passes on the second attempt, sometimes, and CI time doubles. The retry becomes load-bearing, the real cause is buried, and nobody can remove it because removing it makes the build fail.

Or, worse in a security context: a scan that legitimately takes twelve minutes gets killed at ten, the pipeline treats a killed scan as a passed scan, and the gate silently stops gating. That failure has the same shape as the timeout problem and much higher consequences, which is why a killed scan must never be a pass. Fail closed, and make the distinction between "found nothing" and "did not finish" explicit in whatever the pipeline records.

The concession

Sometimes a long-running suite is a design problem and the timeout is correctly telling you so. A 700 second unit test suite usually means the tests are not unit tests: they are touching containers, networks, or real databases, and the right fix is to split the fast ones from the slow ones rather than to raise the number.

So treat a timeout as a prompt to measure, not automatically as a misconfiguration. The measurement tells you which of the two you have, and that is the whole point of taking it.

The implication

Almost every tool in the chain reports a deadline breach and a hang identically, because from the outside the process stopped in both cases. That ambiguity is the defect, not the limit.

Fixing it is cheap: timestamp your output, make progress visible per unit of work, and get a stack dump before the kill. Then the next time this happens the log answers the question in one glance, and nobody spends an afternoon looking for a deadlock that was never there.