On 4 September 2026 function execution stopped across the Inngest platform for 35 minutes. Event ingestion kept working, but some work needed replay. Inngest published its own incident report. All times below are UTC and come from that report.
What happened
- 18:13: function execution became unavailable platform wide.
- 18:17: Inngest declared an incident.
- 18:26: a status page update went out.
- 18:48: connectivity was restored and services began to recover.
- 19:09: the incident moved to monitoring while queued work was processed.
Execution services could not reach the AWS hosted state they need to schedule and run functions. Events were still accepted. Inngest says some affected work may need to be replayed.
The cause, per the vendor
While expanding capacity to a new data center, an engineer changed private network connectivity. AWS allows only one Virtual Private Gateway to be attached to a VPC at a time, so the existing gateway was removed to make room for the new connection.
The team believed that gateway served only an inactive tunnel. It also carried the active Direct Connect association linking the existing data center to the AWS hosted state services. Restoring the VPC attachment did not restore the Direct Connect association, so a second repair was needed.
Inngest names three contributing factors: the network path had no independent redundancy, fallbacks relied on the same unavailable path, and the design created a single network failure domain.
What is confirmed, reported and unknown
- Confirmed by Inngest: the timeline, the removed gateway, the Direct Connect association, the contributing factors and the prevention actions.
- Reported by others: nothing we found. We did not look for independent confirmation.
- Unknown: how many accounts lost work and how many runs needed replay. The report does not give counts.
What to do now
- If you run on Inngest, look for runs that were scheduled on 4 September between 18:13 and 18:48 and confirm they completed.
- Replay anything missing, after checking that completed steps are safe to repeat.
- Map your own private connectivity (VPN, Direct Connect, ExpressRoute, peering) and mark which links are single paths.
- Find any network resource labeled idle, test or legacy, and prove it is idle before deleting it.
How this could have been avoided
- Verify what a resource carries before removing it. Inventory the associations on a gateway, not just its tunnels. Here the active link hid behind an "inactive" label.
- Change review for critical connectivity. Inngest lists planning, validation and review as its fix. A second pair of eyes on a gateway swap is cheap.
- Redundant paths with independent failover. Fallbacks that share the failed path are not fallbacks.
- Test the restore step. Reattaching the gateway did not bring back the Direct Connect association. A runbook that lists every dependent object saves time during the incident.
- Staged changes. Bring the new connection up beside the old one and move traffic across before removing anything.
Try it yourself
Write the list of resources your platform assumes are idle: gateways, load balancers, routes, tunnels. Beside each, write the command or console view that proves nobody is using it.
Click through it at your own pace, or open it full screen.
Quick check
Why did restoring the gateway not fix everything?
The Direct Connect association did not come back with the VPC attachment. The team had to find and restore it separately.
Did customers lose events?
Inngest says events were still accepted. Some work may need replay.
