On 9 October 2026 a data center provider lost connectivity for Inngest, a durable execution and workflow service, and function execution halted for most accounts. On 10 October a second, unrelated failure stalled run scheduling again. Inngest published an incident report on 15 October. Everything below comes from that report unless noted, and all times are UTC.
What happened
9 October, network outage.
- 21:05: a data center provider lost connectivity between its facility and Inngest's other data centers, in both directions, and lost outbound internet access.
- 21:49: connectivity was restored and the queue backlog began to drain.
- 22:26 to 22:38: some internal services that did not recover on their own were restarted.
Inngest quotes the provider's preliminary explanation as "an error during a change to its DDoS protection configuration."
10 October, connection pool collapse.
- 14:00: both PgBouncer poolers lost their database connections and could not rebuild them. Run scheduling and queue processing stalled.
- 14:34 to 14:48: processing began to recover after restarts, which did not hold.
- 15:30: the poolers lost their connections again.
- 16:09 to 16:28: services were moved to poolers running next to the database, and processing recovered.
- 17:10: database error rates returned to normal. Intermittent delays, mostly for cron functions, continued until 04:17 on 11 October.
Root cause, per the vendor
For the 10 October event, Inngest says short client-side deadlines on database lookups cancelled queries that were still waiting in PgBouncer. PgBouncer then ran those abandoned queries and closed connections instead of returning them to the pool. Connections were destroyed faster than they could be replaced. Top-of-hour cron load and a newly rolled out set of lookups started the loop. Inngest says it reproduced the behavior in a test environment.
The report also says both events share a structural cause: function execution depends on three data centers, and a failure on any path between them can interrupt service. Internal failures also counted against configured retries, and a step that failed permanently on an internal error could leave its run in Running, so onFailure handlers and step.invoke callers were never told.
What is confirmed, reported and unknown
- Confirmed by Inngest: the timelines, the pool collapse mechanism (reproduced), the stuck run behavior, and the fixes below.
- Reported, not verified: the data center provider's cause (a DDoS configuration change) is described by Inngest as preliminary. The provider's own report is pending and we have not seen it.
- Unknown: how many accounts and runs were affected. The report says "most accounts" and "some runs." It also does not explain what started the intermittent delays at 21:07 on 10 October.
What to do now
- List every workflow you run on Inngest and find runs still in
Runningfrom 9 to 11 October. - Cancel and replay them. The report warns that replayed runs start from the beginning, so steps that already completed will run again.
- Check that every step with a side effect (charge, email, write) is idempotent before you replay.
- Confirm that your
onFailurehandlers do not assume they will always fire. - Add the vendor status feed to your on call channel.
How this could have been avoided
- Do not cancel work you cannot take back. The pool collapse came from callers abandoning queries that the pooler then executed anyway. Inngest's fix lets a deadline end the caller's wait without cancelling the query.
- Mark permanent failures as failures. Runs left in
Runninghid the problem from customers. Inngest now marks runs failed after internal errors so notifications fire. - Reduce failure domains. Three data centers with dependencies between them meant any link could stop service. Inngest is consolidating into a single cell.
- Customer side: idempotency keys on every side effect. A replay is safe only if repeating a step is safe.
- Customer side: a dead run alarm. An alert on runs older than their expected duration finds stuck work without waiting for a vendor notice.
Try it yourself
Pick one workflow and write down what happens to a customer if its run sits in a running state for six hours. That sentence is your alert threshold.
Click through it at your own pace, or open it full screen.
Quick check
Why are replayed runs risky?
They start again from the beginning, so steps that already completed will run again. Without idempotency, that can double charge or double send.
Has the 9 October network cause been confirmed?
Not fully. Inngest reports the provider's preliminary explanation and says the provider's full report is pending.
