Railway's preliminary incident report, published on 22 October 2026, says an attacker was inside its systems from 20 October 13:58 UTC to 22 October 03:32 UTC. The attacker moved from build infrastructure to CI credentials to decrypted customer environment variables, looking for AI provider API keys. Anyone who stored secrets on Railway before 21 October should rotate them now.
We covered the first disclosure and a full rotation runbook in Railway environment variables breach: rotate your secrets. This post covers what the preliminary report added. That means the minute level timeline, how access moved from one system to the next, what envelope encryption means for working out whether your keys were taken, why someone wants AI keys badly enough to attack a hosting platform, and what any team can learn from it. Everything comes from Railway's public pages or labelled reporting.
What the preliminary report says
Railway published "Preliminary Report: Oct 21st Security Incident" on its engineering blog, written by Jacob Cooper and Angelo Saraceno. The page metadata dates it 22 October 2026 at 23:31 UTC, and the status page linked it at 16:38 UTC on 23 October. Railway calls it preliminary: forensics are ongoing, an external incident response firm and the authorities are involved, and a follow up report is promised when the investigation ends.
The headline facts, as Railway states them:
| Item | What Railway says | Source |
|---|---|---|
| Attack window | 20 Oct 13:58 UTC to 22 Oct 03:32 UTC; the end time is when Railway confirmed the attacker no longer had access to any internal system | Preliminary report |
| When Railway recognised a campaign | 20 Oct 19:53 UTC, described as "a sustained and targeted campaign" | Preliminary report |
| Objective | AI provider API keys, to "perform distillation attacks" | Preliminary report |
| Attacker resources | Access to "abliterated models" and substantial compute, which Railway says enabled highly sophisticated exploits | Preliminary report |
| Upper bound on exposure | At most 1.85% of all workspaces had some environment variables potentially compromised, about 100,000 workspaces | Preliminary report |
| User accounts | Not accessed; individual accounts were not logged into | Preliminary report |
| Source code | Access not ruled out; to be confirmed in the final report | Preliminary report |
| Running workloads | Stayed online; new builds and deployments were paused | Preliminary report |
| Indicators of compromise | None published; Railway withholds exploit details while the investigation runs | Preliminary report |
Railway also says it is not yet able to share "the specific details and scale of the exploits". So the report tells you what was reached and when, but not how the first foothold was gained. That matters for how far you can generalise, and we come back to it under open questions.
Timeline, to the minute
All times are UTC and come from the timeline table in Railway's report unless another source is named. Where the report gives a range, we give the range. Status page entries come from incident QH8L8B3E, "Railway Platform Outage".
| Date | Time (UTC) | Event | Source |
|---|---|---|---|
| 20 Oct | 13:58 | Monitoring pages the on-call engineer for abnormal activity showing up as elevated latency. Research begins. | Report |
| 20 Oct | 14:03 to 16:51 | Investigation identifies an attempted privilege escalation by a hostile external party, with potential escalation in build infrastructure. Railway halts build pipelines. | Report |
| 20 Oct | 16:51 to 18:05 | Mitigations deployed across build systems; access to the affected infrastructure restricted. | Report |
| 20 Oct | 17:03 | Railway declares an incident on status.railway.com. | Report |
| 20 Oct | 18:05 to 19:53 | Engineering reports the exploit remediated and validates that it can no longer be reproduced. | Report |
| 20 Oct | 19:53 | New evidence: the attacker could escalate access at higher scale, across multiple parts of the system, in parallel. | Report |
| 20 Oct | 19:59 to 20:56 | More monitoring and forensics deployed to isolate blast radius and close entry points. | Report |
| 21 Oct | 00:22 to 03:08 | More monitoring and credential restrictions. Further routes of access found. Builds halted again as a precaution. | Report |
| 21 Oct | 04:24 to 05:23 | Forensics show the attacker pulled secrets in encrypted form and could not yet decrypt them. Work starts on revoking access to the systems involved. | Report |
| 21 Oct | 05:23 to 05:27 | Access to the identified systems revoked; non-essential internal systems disabled. | Report |
| 21 Oct | 05:27 to 07:25 | The attacker obtains credentials for Railway's CI environment and decrypts artifacts holding additional API access. | Report |
| 21 Oct | 07:24 | Status incident QH8L8B3E opens: "We are currently investigating this issue". | Status page |
| 21 Oct | 07:25 to 09:18 | Database evidence of an unauthorized query of encrypted environment variables; a subset decrypted using a combination of compromised credentials. | Report |
| 21 Oct | 09:18 to 09:25 | Railway staff read logs from the attacker's payloads showing the specific API key names being searched for. | Report |
| 21 Oct | 09:25 | Railway notifies vendors and asks them to revoke all keys for their service; for some services it hands over every key registered on the platform. (The report's FAQ gives 09:35 for the same step.) | Report |
| 21 Oct | 09:28 | Status page confirms dashboard, API and CLI are down, new builds disabled in all regions, support tooling disrupted. | Report; status page |
| 21 Oct | 20:19 to 22:10 | Session invalidation announced, security disclosure and guidance published, users told about credential rotation. | Report |
| 21 Oct | 21:59 | Status page links the disclosure and tells users to revoke all stored variables, including API keys and private registry credentials. | Status page |
| 21 Oct | 22:10 | Rotation of all storage bucket credentials begins. | Status page |
| 22 Oct | 03:32 | Attacker confirmed out of all internal systems. Builds back for Enterprise workspaces. | Report; status page |
| 22 Oct | 03:32 to 06:29 | Internal systems reinitialised with minimal dependencies, new credentials and extra hardening. Enterprise and Pro builds restored. | Report |
| 22 Oct | 04:17 | All Railway API tokens invalidated. | Status page |
| 22 Oct | 17:00 to 22:49 | Full API restoration; builds for all five plans. Ancillary features return gradually. | Report; status page |
| 22 Oct | 23:31 | Preliminary report published (page metadata). | Railway blog |
| 23 Oct | 16:15 | Railway says it has bucketed customers by risk of exposure and has emailed the highest risk workspaces first. | Status page |
| 23 Oct | 16:38 | Status page links the preliminary report. | Status page |
Two small inconsistencies are worth knowing about. The vendor notification time appears as 09:25 in the timeline and 09:35 in the FAQ. And the report says an incident was declared on the status page at 17:03 on 20 October, while the incident page Railway's disclosure links to (QH8L8B3E) begins at 07:24 on 21 October.
How the attack moved: four stages
Railway has not described the exploits, and neither will we. What the report does give is the order in which systems fell. Read in sequence, it describes a familiar pattern: each stage handed the attacker the credential needed for the next one.
Stage 1: build infrastructure (20 October)
The first signal was latency. That is a useful detail: the earliest symptom was a performance page, not a security alert. Within about five hours the team had identified privilege escalation in build infrastructure, halted builds, mitigated and validated the fix.
Build systems are a natural target on any platform that runs customer code. A builder has to take untrusted source, run arbitrary install scripts and compilers, and produce an image. Railway's own remediation tells you which boundary was in question: it moved builders from its "third generation isolated multi-tenant" design to "fourth generation single-tenant environments", a rollout that was already planned for the following week and was brought forward.
Stage 2: parallel escalation (20 to 21 October)
At 19:53 Railway found that the attacker could escalate "at higher scale, across multiple parts of the Railway system, in-parallel". Railway's report attributes the sophistication to the attacker's compute and models. Over the next nine hours, more routes of access turned up, and builds were halted a second time.
Stage 3: encrypted secrets, then CI credentials (21 October, 04:24 to 07:25)
By 04:24 the attacker had pulled secrets in encrypted form but could not decrypt them. Railway cut access to the systems involved by 05:27. In the same window the attacker reached credentials for Railway's CI environment and decrypted "artifacts that held additional API access".
This is the turning point. Encrypted data is only as safe as the separation between the ciphertext and whatever can unwrap it. CI environments routinely hold deploy credentials, signing material and service tokens, because CI is where software gets built and shipped. A credential found in CI is often the bridge to production.
Stage 4: the database query (21 October, 07:25 to 09:18)
Railway found database evidence of an unauthorized query of encrypted environment variables, "of which a subset were decrypted utilizing a combination of the compromised credentials". Railway then read the attacker's own payload logs and saw which API key names it was searching for. Railway has not published those names. That was the moment it went to vendors.
Why Railway cannot tell you exactly which keys were taken
The most important sentence in the report for customers is a note under the 07:25 entry. Because the full keys were envelope encrypted, the query alone "cannot identify specifically which keys" in an affected project or service were exposed.
Envelope encryption is standard practice. Google Cloud's documentation describes it this way: each piece of data is encrypted with a data encryption key (DEK), and the DEK is encrypted, or wrapped, by a key encryption key (KEK) held in a central key service. One KEK can protect many DEKs, which keeps key management manageable at scale.
The consequence here follows from the design, and this is our reading rather than a Railway statement. A database log shows that rows of ciphertext were read. Whether a given row was then decrypted depends on what happened to the wrapped DEK afterwards, using credentials the attacker held, and that step may leave no per row record where the defender can see it. So Railway knows which workspaces were in the query result, which bounds the exposure at 1.85% of workspaces, but not which individual values were turned back into plaintext.
That is why the guidance is "rotate everything stored before 21 October" rather than "rotate these three keys". It is also why Railway bucketed customers by risk and went to vendors with every key it held for some services.
What 1.85% means in numbers
Railway gives two figures: at most 1.85% of workspaces, and about 100,000 workspaces. Dividing one by the other puts the total near 5.4 million workspaces. That is our arithmetic, not a Railway figure. BleepingComputer reports that Railway has around 4.8 million users and hosts more than 16 million services; users and workspaces are different counts, so the numbers do not conflict.
The group affected by vendor resets is larger than 100,000. Railway says it sent providers "all of the keys we stored, for all users on Railway" for some services, and that a revocation email from a vendor "does not imply" your credential was taken. Railway is also emailing a further group it considers high risk, which it says is larger than 1.85% of workspaces.
Who Railway thinks was targeted
Railway says forensics point to two risk buckets:
- Workspaces with a very large number of environment variables.
- Environments holding many AI keys from vendors.
If either describes you, assume exposure even if you have not had an email.
Why AI API keys, and what "distillation" means
Railway's first disclosure said the attack was part of "a series of attacks recently targeting platform companies with the aim of compromising AI inference providers". The preliminary report is more specific: the attacker was after AI provider keys to run distillation attacks.
Distillation, in Anthropic's words from a February 2026 post on the subject, "involves training a less capable model on the outputs of a stronger one". It is a normal technique when a lab does it to its own models. It becomes an attack when someone harvests a frontier model's outputs at scale, against the provider's terms, to copy its capabilities. Anthropic's post describes such campaigns using networks of fraudulent accounts and resold proxy access so that banning one account does not stop the traffic, and says it built classifiers and behavioural fingerprinting to spot the pattern.
Our reading of why that matters for a hosting platform: a provider can ban a fraudulent account, but a key belonging to a real, paying customer looks like legitimate traffic and bills someone else. Thousands of real customers' keys spread the load across many accounts that each have history and good standing. Railway has not said how it knows the objective beyond the payload logs, and it has not said whether any key was abused.
The report also says the attacker had "abliterated models". Abliteration, as described in a 2024 Hugging Face community post by Maxime Labonne, removes a model's refusal behaviour by finding a "refusal direction" in its internal activations and ablating it, without retraining. In plain terms, an open weights model that will not say no. Railway connects that, plus large compute, to the sophistication of the exploits. It does not say the models wrote the exploits, and we should not assume it.
Separately, The Hacker News quoted a threat research team that recently documented an autonomous agent hijacking exposed AI models. That team told the outlet there is no evidence linking its findings to Railway, while noting that stolen platform keys could be used the same way. We cover the general problem of stolen and abused AI keys in What is LLMjacking, and how do we protect AI API keys?.
Technical details
Railway has published no CVE, no hashes, no attacker IP addresses and no domains. The specifics below are the defender-useful values that do exist in primary sources, with each one attributed.
Incident identifiers and windows
| Field | Value | Source |
|---|---|---|
| Status incident ID | QH8L8B3E ("Railway Platform Outage") | Railway disclosure; status page |
| Attack window | 2026-10-20T13:58Z to 2026-10-22T03:32Z | Preliminary report |
| Encrypted secrets pulled | 2026-10-21T04:24Z to 05:23Z | Preliminary report |
| CI credential use and artifact decryption | 2026-10-21T05:27Z to 07:25Z | Preliminary report |
| Unauthorized variable query and partial decryption | 2026-10-21T07:25Z to 09:18Z | Preliminary report |
| Vendor notification | 09:25Z (timeline) or 09:35Z (FAQ) on 21 Oct | Preliminary report |
| Rotation cutoff for customers | Anything stored before 21 October 2026 | Preliminary report; disclosure |
| Components marked in outage | Builds (US West, EU West, Southeast Asia, US East); API at backboard.railway.com; Support and Community at station.railway.com | Status page |
Credentials Railway rotated or revoked for you
| Credential | What happened | What you must do | Source |
|---|---|---|---|
Storage bucket keys (ACCESS_KEY_ID, SECRET_ACCESS_KEY) | Invalidated and regenerated for every bucket in every environment | Redeploy services that reference them; update anything outside Railway | Bucket and GitHub guide |
| GitHub account connection tokens | Invalidated; all Railway GitHub App installations rolled | Reconnect GitHub; confirm app access | Bucket and GitHub guide; report FAQ |
GitHub tokens stored as registry credentials for ghcr.io | Removed from Railway, but still valid on GitHub | Revoke on GitHub; create a new token with read:packages | Bucket and GitHub guide |
Railway API tokens (RAILWAY_TOKEN project tokens, RAILWAY_API_TOKEN account or workspace tokens) | All invalidated at 04:17 on 22 Oct | Create new tokens with the same scope; update CI secrets | Status page; failing deploys guide |
| CLI sessions and OAuth sessions | Invalidated | railway logout, then railway login | Failing deploys guide; status page |
| Your own variables (database, cloud, AI, payment, registry keys) | Not rotated by Railway; some revoked by vendors | Rotate at the source, update Railway, redeploy | Disclosure; report |
Audit log entries that are expected, not suspicious
Railway's status page and bucket guide say bucket credential revocations were performed by a named Railway engineer and appear in workspace audit logs with the actor type Railway Staff. Railway published the name so customers can tell these events apart from their own team's changes. Any revocation by a staff actor outside that pattern is worth asking Railway about.
Vendors Railway worked with
The report names GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI and Stripe as vendors it worked with to revoke keys, "and others". Its FAQ adds Google to the list of vendors that may have emailed you. Railway says it gave GitHub and SpaceXSI every API key on the platform for their services, whether or not those keys were known to be affected. The status page's guide list adds checks for Grok usage, AWS and GCP.
Reported but unconfirmed by Railway
The Hacker News reported that a security researcher said attackers drained cryptocurrency wallets whose private keys were stored on Railway, naming the receiving address 0xd9773325282874d18cb83f1fe50f14c39750b2ee and a first theft transaction "at 7:36 p.m." (no time zone given), before the official disclosure. Railway's report does not mention wallet theft. Treat the address as a lead for your own investigation, not as a confirmed indicator for this incident.
Error strings that mean "you are still on an old credential"
These come from Railway's recovery guides and are useful for triage after rotation:
| Error | Meaning | Source |
|---|---|---|
InvalidAccessKeyId, SignatureDoesNotMatch | Service still runs the old deployment or holds a pasted old bucket key | Bucket and GitHub guide |
InvalidClientTokenId | Old AWS key still in use | AWS guide |
invalid_grant, PERMISSION_DENIED | Old or disabled Google service account key | GCP guide |
| "unable to pull repository", "build could not be started" | GitHub connection revoked; reconnect | Failing deploys guide |
Who is affected
| Group | Status | What to do |
|---|---|---|
| Workspaces Railway emailed as directly impacted | Variables in the decrypted query set | Rotate every variable today; investigate provider logs from 20 Oct onward |
| Workspaces Railway emailed as high risk | Matches the attacker's targeting pattern | Same as above |
| Any customer who stored variables before 21 Oct | Railway recommends rotation for all | Rotate sensitive variables; prioritise AI, cloud, database and payment keys |
| Customers who got a vendor revocation email | Key revoked as a precaution; not proof of exposure | Deploy the replacement key; still review usage |
Customers using buckets, GitHub deploys, ghcr.io pulls, CLI or CI tokens | Credentials reset by Railway | Redeploy, reconnect, recreate tokens |
| Users of running services only | No downtime for running services | Rotation still recommended |
| New users who signed up after 21 Oct | Outside the rotation guidance | Signups were paused from 22 Oct 19:42 UTC |
Detection: what to check, step by step
Railway did not publish indicators, so detection here means looking for misuse of your own credentials during the window. Use 20 October 2026 13:58 UTC as the start of your search window, not 21 October. That is the most useful single change from the first disclosure.
1. List what you stored, without printing values
Work out which secrets lived on Railway before 21 October. With the CLI linked to each service, you can list variable names without copying values to a shared screen:
railway login
railway link
railway variables --kv | cut -d= -f1 | sort
Railway's documentation notes that sealed variable values are not returned by railway variables or railway run, so check the dashboard for sealed names too. Sealed values are still provided to builds and deployments, so treat them as stored secrets for rotation purposes. Railway has not said whether sealed variables were inside the decrypted subset.
2. AI provider keys first
The report says AI keys were the objective, so start there. For each provider, compare usage by key against normal traffic from 20 October onward, looking for models you never call.
OpenRouter. Railway's guide suggests querying the exposed key before disabling it, because deleting a key removes its usage fields:
curl https://openrouter.ai/api/v1/key \
-H "Authorization: Bearer $EXPOSED_OPENROUTER_KEY"
The response includes usage, usage_daily, usage_weekly, usage_monthly and, for capped keys, limit_remaining. Then disable, export the Activity view as CSV, and create a replacement with a spend limit and an expiry.
Anthropic. Railway's guide points to the Console's Usage, Cost and Logs pages grouped by API key, and to the Usage and Cost Admin API for organisations:
curl "https://api.anthropic.com/v1/organizations/usage_report/messages?starting_at=2026-10-20T00:00:00Z&ending_at=2026-10-23T00:00:00Z&api_key_ids[]=YOUR_KEY_ID&bucket_width=1d" \
-H "anthropic-version: 2023-06-01" \
-H "x-api-key: $ANTHROPIC_ADMIN_KEY"
If an Admin key (prefix sk-ant-admin01-) was stored on Railway, delete it on the Admin keys page as well.
OpenAI. Group usage by API key for the affected project, check billing for the window, and review audit logs and organisation members if enabled. Project keys and legacy user keys live on different pages, so check both.
A model mix that does not match your application, steady high volume at odd hours, or requests in a pattern of many similar prompts are the signals to look for. Volume consistent with harvesting outputs would fit Railway's stated objective, but that is our inference.
3. GitHub
Railway's GitHub guide gives a clear order. Your personal security log keeps 90 days; filter with operation:authentication and operation:access, and look at the oauth_authorization, public_key, personal_access_token and repo categories. For organisations, the audit log keeps 180 days and only owners can read it. Export the filtered log as JSON or CSV before revoking anything, so the evidence survives.
To list GitHub App installations on an organisation from the terminal:
gh api /orgs/YOUR_ORG/installations --jq '.installations[] | [.app_slug, .created_at, .updated_at] | @tsv'
Then, for each repository Railway deployed from, check deploy keys, Actions secrets and variables, webhooks and collaborators. Railway's guide warns that "an attacker with push access" can add a webhook or secret that outlives a password reset. Useful audit log searches include action:integration_installation, action:oauth_application, action:hook and action:personal_access_token.
4. Container registries
If registry credentials lived in Railway variables, review registry activity. Docker's documentation says the Activity page in Docker Home shows the last 7 days by default and keeps 90 days; the Docker Hub API returns the last 3 months, up to 31 days per request. Repository events include Tag Pushed, Tag Deleted and Privacy Changed; organisation events include Access token created, updated, revoked and deleted. Export to CSV with the window filter applied. An unexpected Tag Pushed on an image you deploy is the event that matters most, because it puts attacker code into your next deploy.
For ghcr.io, revoke the old personal access token on GitHub. Removing it from Railway did not kill it.
5. Cloud accounts
AWS. Event history is per Region, so search each one:
for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
aws cloudtrail lookup-events --region "$r" \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLEKEYID \
--start-time 2026-10-20T13:00:00Z --max-results 50 \
--query 'Events[].[EventTime,EventName,EventSource]' --output text
done
Railway's AWS guide flags RunInstances, CreateUser, CreateAccessKey, AttachUserPolicy, CreateRole and PutBucketPolicy, plus any activity in Regions you do not use.
Google Cloud. Railway's GCP guide notes that service account keys do not expire by default, and that tokens already minted from a disabled key stay valid for up to an hour. Disable the key, and disable the account itself if you need access cut immediately:
gcloud iam service-accounts keys disable KEY_ID \
--iam-account=SA_NAME@PROJECT.iam.gserviceaccount.com --project=PROJECT
gcloud policy-intelligence query-activity \
--activity-type=serviceAccountKeyLastAuthentication --project=PROJECT
In Logs Explorer, query logName:"cloudaudit.googleapis.com" with protoPayload.authenticationInfo.principalEmail set to the exposed account, and check for impersonation through serviceAccountDelegationInfo. Data Access logs are off by default except for BigQuery, so reads of your data may not be logged at all. Compare billing by SKU for the window; the guide names compute, GPU and Vertex AI as the common targets.
6. Databases
Railway asks every customer to remove public network access to databases. If a database had a public TCP proxy and its connection string was in a variable, assume someone may have connected. Review connection logs if your engine keeps them, rotate the password, and move clients to private networking.
-- PostgreSQL: rotate the application role's password
ALTER ROLE app_user WITH PASSWORD 'new-long-random-value';
-- Check who is connected right now
SELECT usename, client_addr, backend_start FROM pg_stat_activity WHERE usename = 'app_user';
7. Payments and signing secrets
Stripe keys, webhook signing secrets and JWT or session signing keys count too. Signing keys never appear in a usage graph, so they get skipped; rotate them and accept that existing sessions are invalidated.
Remediation: the order that avoids re-exposure
Railway explained one of its own decisions in a way every team can reuse. It waited until the attacker was out before telling users to roll credentials, "to prevent the credentials from becoming immediately compromised again". The attacker was confirmed out at 03:32 on 22 October, so rotations you did before that time should be done again. Rotations done on 21 October before the disclosure went out are the ones most likely to need a second pass.
The order:
- Rotate at the source. Create the new key at the provider, scoped as narrowly as the provider allows, with a spend cap and expiry where supported.
- Update Railway. Edit the variable, or the shared variable in Project Settings, so every reference picks it up.
- Redeploy. Containers read variables at start. Use Deploy Latest Commit from the command palette or
railway redeploy --service NAME. Railway notes that redeploying a deployment from before the incident fails; deploy the latest commit instead. - Revoke the old key. Only after the new deployment is healthy, so you do not take yourself down.
- Recreate automation credentials and reconnect GitHub.
railway logoutthenrailway loginfor the CLI, newRAILWAY_TOKENorRAILWAY_API_TOKENvalues in CI secrets, and a fresh GitHub connection. If you want your own clean point for buckets,railway bucket credentials --resetregenerates them. - Write it down. Keep exported logs and a timestamped list of rotated keys. Railway's audit log retention depends on plan, so export now.
Our earlier post has a longer rotation runbook per credential type in What to do now: a rotation runbook.
What changed since the first disclosure
| Topic | 21 Oct disclosure | 22 Oct preliminary report |
|---|---|---|
| Start of activity | "Earlier this morning" (21 Oct) | 20 Oct 13:58 UTC |
| Entry point | Systems that store environment variables | Build infrastructure first, then CI credentials, then the variables database |
| Objective | Part of attacks aimed at compromising AI inference providers | AI provider keys for distillation attacks |
| Scope | "A subset of users" | At most 1.85% of workspaces, about 100,000 |
| Which keys | Not stated | Cannot be identified individually because of envelope encryption |
| User accounts | Not stated | Not accessed |
| Source code | Not stated | Not ruled out |
| Vendors | Upstream providers notified | GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe and others named |
| Builds | All paused | Restored for all tiers; builders moved to single-tenant |
| Attacker status | Not stated | Confirmed out at 22 Oct 03:32 UTC |
Open questions
- Initial access. How did the attacker first escalate in build infrastructure? Railway withholds this for now. Until it is published, nobody outside can say whether the weakness was specific to Railway or shared with other platforms.
- Which key names were searched for. Railway read them in the payload logs but has not published them. They would help customers prioritise.
- Sealed variables. Were sealed values in the decrypted subset? Not stated.
- Source code. Railway will confirm in the final report.
- Abuse. Railway has not said whether any key was actually used. The wallet theft claim comes from a researcher quoted by The Hacker News and is unconfirmed by Railway.
- Attribution. None offered. Railway says authorities are involved.
- The 17:03 status entry. The report says an incident was declared on 20 October; the public incident
QH8L8B3Estarts on 21 October.
What would have helped
Nobody outside Railway can say what would have stopped this specific attack, because the exploit details are not public. What the report does show is how a single foothold turned into exposed customer secrets, and every stage has a general lesson.
Practices any team can use
If you store secrets on a hosting platform:
- Know where every secret lives. A list of secret names, owners, scope and the services that use them turns a breach day into a checklist. Without one, you are reading dashboards one at a time.
- Prefer short lived credentials. Workload identity federation for cloud access, OIDC from CI to cloud, and tokens that expire in minutes leave nothing durable to steal. A static key that "doesn't expire by default", in Railway's description of Google keys, is the opposite.
- Cap and expire AI keys. Separate keys per service and environment, spend limits and expiry dates where the provider supports them. A capped key turns abuse into a small bill and an alert.
- Use static outbound IPs where you can. Railway's GCP guide points out that with static egress, calls from any other address point at an attacker; with shared egress, an unfamiliar IP is only a lead.
- Keep databases off the public internet. Private networking means a leaked connection string is not enough on its own.
- Export logs before you need them. GitHub organisation logs keep 180 days, Docker Home 90, AWS CloudTrail event history 90. Send them somewhere you control.
- Rehearse rotation. Rotating dozens of keys across providers for the first time, while builds are paused, is slow and error prone.
If you run a platform, a CI system or anything that builds other people's code:
- Treat builders as hostile. Customer code runs there; Railway's move to single-tenant builders points the same way.
- Keep CI away from decryption authority. CI credentials here led to decrypted artifacts with more API access. Scope CI secrets to the job that needs them.
- Log every unwrap. If each data key decryption is a logged call to a key service with per object context, you can tell customers exactly which secrets were decrypted.
- Build a kill switch and a mass revoke before you need them, as Railway did by pausing builds while running services stayed up.
- Page on anomalies, not just alerts. Railway's first signal was elevated latency.
Where Safeguard fits
Be clear about the limits first. This incident happened inside a hosting provider's build systems, CI environment and secrets database. Safeguard does not run inside your provider, does not hold or rotate your provider keys, and cannot see whether a variable on someone else's platform was decrypted. We do not claim it would have prevented or detected this breach.
Where Safeguard does help is on your side of the line, in the code and images you build and ship:
- Knowing what you run. SBOM and dependency scanning of repositories, container images and SBOM files (
safeguard scan --dir .,safeguard scan --image <name>,safeguard scan --sbom <file>, with JSON results under./output). The platform shows which projects contain a given component and version. After a provider breach, the follow on question is often which of your services use a particular SDK or client library to talk to that provider; that becomes a query. - Keeping dependencies current. Vulnerability matching against vulnerability databases, with reachability analysis to rank the code paths your application actually calls.
- Stopping regressions. Policies and gates evaluated in the platform at build, CI and release.
- Approved changes during a messy week. Rotation day means many small code and configuration edits: moving a hard coded client setup to read from a variable, adding a key ID to logs, switching a library to workload identity. The Safeguard CLI's assistant can make those edits and run commands with your approval, and
/permissionsoffers Ask, Accept edits, Plan, Auto (the default) and Strict.safeguard execruns in CI. The desktop and phone apps can send work to a computer running the CLI and approve each command. - Fix pull requests, off by default. Autonomous remediation opens fix pull requests for vulnerable dependencies and ships with every category set to Manual.
Zero-Day Discovery and Gold
There is no CVE in Railway's report and no known vulnerable open source package. Zero-Day Discovery looks for previously unknown flaws before a CVE exists, in an agentic mode and a deterministic scanner mode, issues findings as SGZ advisories before they reach public CVE databases, and covers seven package ecosystems. A hosting provider's internal build and CI systems are outside what it looks at, and we make no claim that it found anything related to this incident.
Gold Open Source (gold.safeguard.sh) is free CVE intelligence with CVSS, EPSS, KEV status, affected and fixed versions and a recommended safe version, plus a free read API and a CI gate GitHub Action. Gold Registry (registry.safeguard.sh) supplies hardened, Sigstore signed components and container images held to zero exploitable critical or high CVEs with SLSA L3 or higher provenance. Neither has a record tied to this incident. The habit they support is the right one after any platform breach: when you redeploy on fresh credentials, rebuild from patched, signed components rather than reusing whatever image was cached.
Common mistakes
- Starting the search on 21 October. The report puts the first activity at 13:58 UTC on 20 October.
- Rotating before the attacker was out. Rotations finished before 03:32 UTC on 22 October may need a second pass, by the same logic Railway used for its own keys.
- Assuming a vendor revocation email means you were breached, or that no email means you were not. Railway says vendor resets went wider than the exposed set, and it cannot tell which values were decrypted.
- Deleting a key before reading its usage. Railway's OpenRouter guide notes that usage fields disappear with a deleted key. Read, export, then delete.
- Forgetting the token still works elsewhere. GitHub tokens removed from Railway's registry settings still work on GitHub until you revoke them there.
- Treating sealed as safe. Sealed values are hidden from the UI and API but still delivered to builds and deployments. Rotate them too.
FAQ
What is Railway's preliminary incident report?
It is Railway's 22 October 2026 follow up to its 21 October security disclosure. It gives a UTC timeline of an attack from 20 October 13:58 to 22 October 03:32, says the attacker was after AI provider API keys, bounds exposure at 1.85% of workspaces and lists Railway's fixes. A final forensic report is promised.
How did the attacker get from builds to my secrets?
Railway describes the order, not the exploits: privilege escalation in build infrastructure, wider parallel escalation, a pull of encrypted secrets, then credentials for Railway's CI environment that allowed decryption of artifacts holding more API access, and finally a query of encrypted environment variables of which a subset was decrypted.
Why can't Railway tell me whether my specific key was decrypted?
The variables were envelope encrypted. Railway can see that encrypted rows were queried but says the query alone cannot identify which keys were decrypted. So it recommends rotating everything stored before 21 October and is contacting customers by risk.
What is a distillation attack?
Training a weaker model on a stronger model's outputs, done at scale against a provider's terms to copy its capabilities. Anthropic described such campaigns in February 2026. Stolen customer keys let an attacker spread that traffic across real, paying accounts that are harder to ban.
Was my source code accessed?
Railway has not ruled it out and says it will confirm in the final report. If your repositories contain anything sensitive, treat this as open.
I got a revocation email from GitHub, OpenAI or another vendor. Was my key stolen?
Not necessarily. Railway sent some vendors every key it held for their service, and says a vendor email does not imply your credential was affected. Deploy the replacement and review usage anyway.
Did attackers steal cryptocurrency?
The Hacker News quoted a researcher who said wallets with private keys stored on Railway were drained and named a receiving address. Railway's report does not mention it. If you stored wallet keys on Railway, move funds to a new wallet whose key has never been on the platform.
Is Railway safe to use now?
Railway says it has revalidated all impacted systems, rebuilt tooling from validated source, moved builders to single-tenant environments and rolled its own keys twice. The rotation steps apply either way, and so do the lessons for any platform that stores your secrets, as the Zeabur shutdown showed.
Fix it without waiting for upstream
Railway's breach is not a dependency vulnerability, and nothing below rotates a key. But a redeploy across every service is exactly the moment to make sure what you ship is current, so the same week fixes both.
Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.
1. Autonomous remediation: pull requests, not tickets
When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.
2. Gold Open Source: the safe version in seconds, free
gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.
3. Gold Registry: start from components that are already fixed
registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.
When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.
| Your situation | What to use |
|---|---|
| A fixed version exists | Autonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version |
| You need to know today whether you are affected | Gold Open Source lookup and "Am I affected?" checker, free and without login |
| You want vulnerable versions to stop shipping | The Gold scan GitHub Action in CI, plus Safeguard policies and gates |
| Upstream has no fix yet, or will not ship one | A custom Gold request; fork and patch (early access) |
| You want to start clean on new projects | Pull hardened components and images from Gold Registry |
One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.
See how Safeguard would have helped
These are real product sessions on other examples. None of them shows Railway, this breach or any customer's keys, and none of them rotates a credential at a provider for you.
The CLI demo shows the assistant fixing a lodash CVE: it reads the finding and advisory, asks before each command, edits package.json, runs install and tests with your approval, prints a summary and pushes nothing. For Railway, you would open the CLI in a service's repository and ask it to move a hard coded AI client setup to read its key from an environment variable, or to bump the base image before you redeploy on new credentials. Every command and edit waits for your approval.
Click through it at your own pace, or open it full screen.
The desktop app demo shows a CLI session in the conversation list, a folder choice, and Approve or Deny on each command and file edit, ending with the upgrade done and tests passing. Here you would pick the repository behind a Railway service, ask for the change, and approve each edit yourself before running railway up or pushing for an autodeploy.
Click through it at your own pace, or open it full screen.
The mobile demo shows a real Android session: work sent to a computer running the CLI, and the command approved on the phone with Allow once, Allow for turn or Deny. On a rotation day spread across many services, that lets the person on call approve each scripted step from wherever they are, without handing the assistant open ended permission.
Click through it at your own pace, or open it full screen.
To try it on your own systems, download Safeguard and read the CLI documentation. For related reading, see are environment variables on hosting platforms safe for secrets?, the LiteLLM post-mortem on agent tokens in gateway logs and October 2026 security incidents and what would have helped.
Quick check
When should your log search for the Railway incident start?
At 13:58 UTC on 20 October 2026, when Railway's report says the attack window opened. The first disclosure only said "earlier this morning" on 21 October, so searches based on it start too late.
Why does Railway recommend rotating every variable instead of naming the exposed ones?
The variables were envelope encrypted, and Railway says the attacker's query alone cannot show which keys were decrypted. It can bound the exposure at 1.85% of workspaces, but not identify individual values, so every secret stored before 21 October is treated as exposed.
Sources
- Railway: Preliminary Report: Oct 21st Security Incident (primary)
- Railway: Preliminary Report, Markdown version (primary)
- Railway: Security Disclosure: Wednesday October 21st (primary)
- Railway status: Railway Platform Outage, incident QH8L8B3E (primary)
- Railway: Recover Failing Deploys (primary)
- Railway: Redeploy Buckets and Reconnect GitHub After the Credential Reset (primary)
- Railway: Check for Anomalous GitHub Access (primary)
- Railway: Check for Anomalous GCP Access (primary)
- Railway: Check for Anomalous AWS Access (primary)
- Railway: Check for Unexpected Anthropic Usage (primary)
- Railway: Check for Unexpected OpenAI Usage (primary)
- Railway: Check for Unexpected OpenRouter Usage (primary)
- Railway docs: Variables (including sealed variables)
- GitHub Docs: Reviewing the audit log for your organization
- Docker Docs: Activity logs
- Google Cloud: Envelope encryption
- Anthropic: Detecting and preventing distillation attacks
- Hugging Face: Uncensor any LLM with abliteration
- BleepingComputer: Railway warns devs to rotate secrets after potential breach
- The Hacker News: Railway says intruder may have accessed customer secrets
- Hacker News discussion: Preliminary Report Oct 21st Security Incident
- Safeguard: Railway environment variables breach (October 2026)
Last reviewed: 23 October 2026. This is a developing story; check the linked advisories for the latest status.