Safeguard
Supply Chain Security•

Railway Preliminary Incident Report: CI Credentials and AI Key Theft

Railway's preliminary report: a 20 to 22 Oct breach moved from build systems to CI credentials and decrypted variables, hunting AI API keys. What to do.

Safeguard Security Team
39 min read

Railway's preliminary incident report, published on 22 October 2026, says an attacker was inside its systems from 20 October 13:58 UTC to 22 October 03:32 UTC. The attacker moved from build infrastructure to CI credentials to decrypted customer environment variables, looking for AI provider API keys. Anyone who stored secrets on Railway before 21 October should rotate them now.

We covered the first disclosure and a full rotation runbook in Railway environment variables breach: rotate your secrets. This post covers what the preliminary report added. That means the minute level timeline, how access moved from one system to the next, what envelope encryption means for working out whether your keys were taken, why someone wants AI keys badly enough to attack a hosting platform, and what any team can learn from it. Everything comes from Railway's public pages or labelled reporting.

What the preliminary report says

Railway published "Preliminary Report: Oct 21st Security Incident" on its engineering blog, written by Jacob Cooper and Angelo Saraceno. The page metadata dates it 22 October 2026 at 23:31 UTC, and the status page linked it at 16:38 UTC on 23 October. Railway calls it preliminary: forensics are ongoing, an external incident response firm and the authorities are involved, and a follow up report is promised when the investigation ends.

The headline facts, as Railway states them:

ItemWhat Railway saysSource
Attack window20 Oct 13:58 UTC to 22 Oct 03:32 UTC; the end time is when Railway confirmed the attacker no longer had access to any internal systemPreliminary report
When Railway recognised a campaign20 Oct 19:53 UTC, described as "a sustained and targeted campaign"Preliminary report
ObjectiveAI provider API keys, to "perform distillation attacks"Preliminary report
Attacker resourcesAccess to "abliterated models" and substantial compute, which Railway says enabled highly sophisticated exploitsPreliminary report
Upper bound on exposureAt most 1.85% of all workspaces had some environment variables potentially compromised, about 100,000 workspacesPreliminary report
User accountsNot accessed; individual accounts were not logged intoPreliminary report
Source codeAccess not ruled out; to be confirmed in the final reportPreliminary report
Running workloadsStayed online; new builds and deployments were pausedPreliminary report
Indicators of compromiseNone published; Railway withholds exploit details while the investigation runsPreliminary report

Railway also says it is not yet able to share "the specific details and scale of the exploits". So the report tells you what was reached and when, but not how the first foothold was gained. That matters for how far you can generalise, and we come back to it under open questions.

Timeline, to the minute

All times are UTC and come from the timeline table in Railway's report unless another source is named. Where the report gives a range, we give the range. Status page entries come from incident QH8L8B3E, "Railway Platform Outage".

DateTime (UTC)EventSource
20 Oct13:58Monitoring pages the on-call engineer for abnormal activity showing up as elevated latency. Research begins.Report
20 Oct14:03 to 16:51Investigation identifies an attempted privilege escalation by a hostile external party, with potential escalation in build infrastructure. Railway halts build pipelines.Report
20 Oct16:51 to 18:05Mitigations deployed across build systems; access to the affected infrastructure restricted.Report
20 Oct17:03Railway declares an incident on status.railway.com.Report
20 Oct18:05 to 19:53Engineering reports the exploit remediated and validates that it can no longer be reproduced.Report
20 Oct19:53New evidence: the attacker could escalate access at higher scale, across multiple parts of the system, in parallel.Report
20 Oct19:59 to 20:56More monitoring and forensics deployed to isolate blast radius and close entry points.Report
21 Oct00:22 to 03:08More monitoring and credential restrictions. Further routes of access found. Builds halted again as a precaution.Report
21 Oct04:24 to 05:23Forensics show the attacker pulled secrets in encrypted form and could not yet decrypt them. Work starts on revoking access to the systems involved.Report
21 Oct05:23 to 05:27Access to the identified systems revoked; non-essential internal systems disabled.Report
21 Oct05:27 to 07:25The attacker obtains credentials for Railway's CI environment and decrypts artifacts holding additional API access.Report
21 Oct07:24Status incident QH8L8B3E opens: "We are currently investigating this issue".Status page
21 Oct07:25 to 09:18Database evidence of an unauthorized query of encrypted environment variables; a subset decrypted using a combination of compromised credentials.Report
21 Oct09:18 to 09:25Railway staff read logs from the attacker's payloads showing the specific API key names being searched for.Report
21 Oct09:25Railway notifies vendors and asks them to revoke all keys for their service; for some services it hands over every key registered on the platform. (The report's FAQ gives 09:35 for the same step.)Report
21 Oct09:28Status page confirms dashboard, API and CLI are down, new builds disabled in all regions, support tooling disrupted.Report; status page
21 Oct20:19 to 22:10Session invalidation announced, security disclosure and guidance published, users told about credential rotation.Report
21 Oct21:59Status page links the disclosure and tells users to revoke all stored variables, including API keys and private registry credentials.Status page
21 Oct22:10Rotation of all storage bucket credentials begins.Status page
22 Oct03:32Attacker confirmed out of all internal systems. Builds back for Enterprise workspaces.Report; status page
22 Oct03:32 to 06:29Internal systems reinitialised with minimal dependencies, new credentials and extra hardening. Enterprise and Pro builds restored.Report
22 Oct04:17All Railway API tokens invalidated.Status page
22 Oct17:00 to 22:49Full API restoration; builds for all five plans. Ancillary features return gradually.Report; status page
22 Oct23:31Preliminary report published (page metadata).Railway blog
23 Oct16:15Railway says it has bucketed customers by risk of exposure and has emailed the highest risk workspaces first.Status page
23 Oct16:38Status page links the preliminary report.Status page

Two small inconsistencies are worth knowing about. The vendor notification time appears as 09:25 in the timeline and 09:35 in the FAQ. And the report says an incident was declared on the status page at 17:03 on 20 October, while the incident page Railway's disclosure links to (QH8L8B3E) begins at 07:24 on 21 October.

How the attack moved: four stages

Railway has not described the exploits, and neither will we. What the report does give is the order in which systems fell. Read in sequence, it describes a familiar pattern: each stage handed the attacker the credential needed for the next one.

Stage 1: build infrastructure (20 October)

The first signal was latency. That is a useful detail: the earliest symptom was a performance page, not a security alert. Within about five hours the team had identified privilege escalation in build infrastructure, halted builds, mitigated and validated the fix.

Build systems are a natural target on any platform that runs customer code. A builder has to take untrusted source, run arbitrary install scripts and compilers, and produce an image. Railway's own remediation tells you which boundary was in question: it moved builders from its "third generation isolated multi-tenant" design to "fourth generation single-tenant environments", a rollout that was already planned for the following week and was brought forward.

Stage 2: parallel escalation (20 to 21 October)

At 19:53 Railway found that the attacker could escalate "at higher scale, across multiple parts of the Railway system, in-parallel". Railway's report attributes the sophistication to the attacker's compute and models. Over the next nine hours, more routes of access turned up, and builds were halted a second time.

Stage 3: encrypted secrets, then CI credentials (21 October, 04:24 to 07:25)

By 04:24 the attacker had pulled secrets in encrypted form but could not decrypt them. Railway cut access to the systems involved by 05:27. In the same window the attacker reached credentials for Railway's CI environment and decrypted "artifacts that held additional API access".

This is the turning point. Encrypted data is only as safe as the separation between the ciphertext and whatever can unwrap it. CI environments routinely hold deploy credentials, signing material and service tokens, because CI is where software gets built and shipped. A credential found in CI is often the bridge to production.

Stage 4: the database query (21 October, 07:25 to 09:18)

Railway found database evidence of an unauthorized query of encrypted environment variables, "of which a subset were decrypted utilizing a combination of the compromised credentials". Railway then read the attacker's own payload logs and saw which API key names it was searching for. Railway has not published those names. That was the moment it went to vendors.

Why Railway cannot tell you exactly which keys were taken

The most important sentence in the report for customers is a note under the 07:25 entry. Because the full keys were envelope encrypted, the query alone "cannot identify specifically which keys" in an affected project or service were exposed.

Envelope encryption is standard practice. Google Cloud's documentation describes it this way: each piece of data is encrypted with a data encryption key (DEK), and the DEK is encrypted, or wrapped, by a key encryption key (KEK) held in a central key service. One KEK can protect many DEKs, which keeps key management manageable at scale.

The consequence here follows from the design, and this is our reading rather than a Railway statement. A database log shows that rows of ciphertext were read. Whether a given row was then decrypted depends on what happened to the wrapped DEK afterwards, using credentials the attacker held, and that step may leave no per row record where the defender can see it. So Railway knows which workspaces were in the query result, which bounds the exposure at 1.85% of workspaces, but not which individual values were turned back into plaintext.

That is why the guidance is "rotate everything stored before 21 October" rather than "rotate these three keys". It is also why Railway bucketed customers by risk and went to vendors with every key it held for some services.

What 1.85% means in numbers

Railway gives two figures: at most 1.85% of workspaces, and about 100,000 workspaces. Dividing one by the other puts the total near 5.4 million workspaces. That is our arithmetic, not a Railway figure. BleepingComputer reports that Railway has around 4.8 million users and hosts more than 16 million services; users and workspaces are different counts, so the numbers do not conflict.

The group affected by vendor resets is larger than 100,000. Railway says it sent providers "all of the keys we stored, for all users on Railway" for some services, and that a revocation email from a vendor "does not imply" your credential was taken. Railway is also emailing a further group it considers high risk, which it says is larger than 1.85% of workspaces.

Who Railway thinks was targeted

Railway says forensics point to two risk buckets:

  1. Workspaces with a very large number of environment variables.
  2. Environments holding many AI keys from vendors.

If either describes you, assume exposure even if you have not had an email.

Why AI API keys, and what "distillation" means

Railway's first disclosure said the attack was part of "a series of attacks recently targeting platform companies with the aim of compromising AI inference providers". The preliminary report is more specific: the attacker was after AI provider keys to run distillation attacks.

Distillation, in Anthropic's words from a February 2026 post on the subject, "involves training a less capable model on the outputs of a stronger one". It is a normal technique when a lab does it to its own models. It becomes an attack when someone harvests a frontier model's outputs at scale, against the provider's terms, to copy its capabilities. Anthropic's post describes such campaigns using networks of fraudulent accounts and resold proxy access so that banning one account does not stop the traffic, and says it built classifiers and behavioural fingerprinting to spot the pattern.

Our reading of why that matters for a hosting platform: a provider can ban a fraudulent account, but a key belonging to a real, paying customer looks like legitimate traffic and bills someone else. Thousands of real customers' keys spread the load across many accounts that each have history and good standing. Railway has not said how it knows the objective beyond the payload logs, and it has not said whether any key was abused.

The report also says the attacker had "abliterated models". Abliteration, as described in a 2024 Hugging Face community post by Maxime Labonne, removes a model's refusal behaviour by finding a "refusal direction" in its internal activations and ablating it, without retraining. In plain terms, an open weights model that will not say no. Railway connects that, plus large compute, to the sophistication of the exploits. It does not say the models wrote the exploits, and we should not assume it.

Separately, The Hacker News quoted a threat research team that recently documented an autonomous agent hijacking exposed AI models. That team told the outlet there is no evidence linking its findings to Railway, while noting that stolen platform keys could be used the same way. We cover the general problem of stolen and abused AI keys in What is LLMjacking, and how do we protect AI API keys?.

Technical details

Railway has published no CVE, no hashes, no attacker IP addresses and no domains. The specifics below are the defender-useful values that do exist in primary sources, with each one attributed.

Incident identifiers and windows

FieldValueSource
Status incident IDQH8L8B3E ("Railway Platform Outage")Railway disclosure; status page
Attack window2026-10-20T13:58Z to 2026-10-22T03:32ZPreliminary report
Encrypted secrets pulled2026-10-21T04:24Z to 05:23ZPreliminary report
CI credential use and artifact decryption2026-10-21T05:27Z to 07:25ZPreliminary report
Unauthorized variable query and partial decryption2026-10-21T07:25Z to 09:18ZPreliminary report
Vendor notification09:25Z (timeline) or 09:35Z (FAQ) on 21 OctPreliminary report
Rotation cutoff for customersAnything stored before 21 October 2026Preliminary report; disclosure
Components marked in outageBuilds (US West, EU West, Southeast Asia, US East); API at backboard.railway.com; Support and Community at station.railway.comStatus page

Credentials Railway rotated or revoked for you

CredentialWhat happenedWhat you must doSource
Storage bucket keys (ACCESS_KEY_ID, SECRET_ACCESS_KEY)Invalidated and regenerated for every bucket in every environmentRedeploy services that reference them; update anything outside RailwayBucket and GitHub guide
GitHub account connection tokensInvalidated; all Railway GitHub App installations rolledReconnect GitHub; confirm app accessBucket and GitHub guide; report FAQ
GitHub tokens stored as registry credentials for ghcr.ioRemoved from Railway, but still valid on GitHubRevoke on GitHub; create a new token with read:packagesBucket and GitHub guide
Railway API tokens (RAILWAY_TOKEN project tokens, RAILWAY_API_TOKEN account or workspace tokens)All invalidated at 04:17 on 22 OctCreate new tokens with the same scope; update CI secretsStatus page; failing deploys guide
CLI sessions and OAuth sessionsInvalidatedrailway logout, then railway loginFailing deploys guide; status page
Your own variables (database, cloud, AI, payment, registry keys)Not rotated by Railway; some revoked by vendorsRotate at the source, update Railway, redeployDisclosure; report

Audit log entries that are expected, not suspicious

Railway's status page and bucket guide say bucket credential revocations were performed by a named Railway engineer and appear in workspace audit logs with the actor type Railway Staff. Railway published the name so customers can tell these events apart from their own team's changes. Any revocation by a staff actor outside that pattern is worth asking Railway about.

Vendors Railway worked with

The report names GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI and Stripe as vendors it worked with to revoke keys, "and others". Its FAQ adds Google to the list of vendors that may have emailed you. Railway says it gave GitHub and SpaceXSI every API key on the platform for their services, whether or not those keys were known to be affected. The status page's guide list adds checks for Grok usage, AWS and GCP.

Reported but unconfirmed by Railway

The Hacker News reported that a security researcher said attackers drained cryptocurrency wallets whose private keys were stored on Railway, naming the receiving address 0xd9773325282874d18cb83f1fe50f14c39750b2ee and a first theft transaction "at 7:36 p.m." (no time zone given), before the official disclosure. Railway's report does not mention wallet theft. Treat the address as a lead for your own investigation, not as a confirmed indicator for this incident.

Error strings that mean "you are still on an old credential"

These come from Railway's recovery guides and are useful for triage after rotation:

ErrorMeaningSource
InvalidAccessKeyId, SignatureDoesNotMatchService still runs the old deployment or holds a pasted old bucket keyBucket and GitHub guide
InvalidClientTokenIdOld AWS key still in useAWS guide
invalid_grant, PERMISSION_DENIEDOld or disabled Google service account keyGCP guide
"unable to pull repository", "build could not be started"GitHub connection revoked; reconnectFailing deploys guide

Who is affected

GroupStatusWhat to do
Workspaces Railway emailed as directly impactedVariables in the decrypted query setRotate every variable today; investigate provider logs from 20 Oct onward
Workspaces Railway emailed as high riskMatches the attacker's targeting patternSame as above
Any customer who stored variables before 21 OctRailway recommends rotation for allRotate sensitive variables; prioritise AI, cloud, database and payment keys
Customers who got a vendor revocation emailKey revoked as a precaution; not proof of exposureDeploy the replacement key; still review usage
Customers using buckets, GitHub deploys, ghcr.io pulls, CLI or CI tokensCredentials reset by RailwayRedeploy, reconnect, recreate tokens
Users of running services onlyNo downtime for running servicesRotation still recommended
New users who signed up after 21 OctOutside the rotation guidanceSignups were paused from 22 Oct 19:42 UTC

Detection: what to check, step by step

Railway did not publish indicators, so detection here means looking for misuse of your own credentials during the window. Use 20 October 2026 13:58 UTC as the start of your search window, not 21 October. That is the most useful single change from the first disclosure.

1. List what you stored, without printing values

Work out which secrets lived on Railway before 21 October. With the CLI linked to each service, you can list variable names without copying values to a shared screen:

railway login
railway link
railway variables --kv | cut -d= -f1 | sort

Railway's documentation notes that sealed variable values are not returned by railway variables or railway run, so check the dashboard for sealed names too. Sealed values are still provided to builds and deployments, so treat them as stored secrets for rotation purposes. Railway has not said whether sealed variables were inside the decrypted subset.

2. AI provider keys first

The report says AI keys were the objective, so start there. For each provider, compare usage by key against normal traffic from 20 October onward, looking for models you never call.

OpenRouter. Railway's guide suggests querying the exposed key before disabling it, because deleting a key removes its usage fields:

curl https://openrouter.ai/api/v1/key \
  -H "Authorization: Bearer $EXPOSED_OPENROUTER_KEY"

The response includes usage, usage_daily, usage_weekly, usage_monthly and, for capped keys, limit_remaining. Then disable, export the Activity view as CSV, and create a replacement with a spend limit and an expiry.

Anthropic. Railway's guide points to the Console's Usage, Cost and Logs pages grouped by API key, and to the Usage and Cost Admin API for organisations:

curl "https://api.anthropic.com/v1/organizations/usage_report/messages?starting_at=2026-10-20T00:00:00Z&ending_at=2026-10-23T00:00:00Z&api_key_ids[]=YOUR_KEY_ID&bucket_width=1d" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-api-key: $ANTHROPIC_ADMIN_KEY"

If an Admin key (prefix sk-ant-admin01-) was stored on Railway, delete it on the Admin keys page as well.

OpenAI. Group usage by API key for the affected project, check billing for the window, and review audit logs and organisation members if enabled. Project keys and legacy user keys live on different pages, so check both.

A model mix that does not match your application, steady high volume at odd hours, or requests in a pattern of many similar prompts are the signals to look for. Volume consistent with harvesting outputs would fit Railway's stated objective, but that is our inference.

3. GitHub

Railway's GitHub guide gives a clear order. Your personal security log keeps 90 days; filter with operation:authentication and operation:access, and look at the oauth_authorization, public_key, personal_access_token and repo categories. For organisations, the audit log keeps 180 days and only owners can read it. Export the filtered log as JSON or CSV before revoking anything, so the evidence survives.

To list GitHub App installations on an organisation from the terminal:

gh api /orgs/YOUR_ORG/installations --jq '.installations[] | [.app_slug, .created_at, .updated_at] | @tsv'

Then, for each repository Railway deployed from, check deploy keys, Actions secrets and variables, webhooks and collaborators. Railway's guide warns that "an attacker with push access" can add a webhook or secret that outlives a password reset. Useful audit log searches include action:integration_installation, action:oauth_application, action:hook and action:personal_access_token.

4. Container registries

If registry credentials lived in Railway variables, review registry activity. Docker's documentation says the Activity page in Docker Home shows the last 7 days by default and keeps 90 days; the Docker Hub API returns the last 3 months, up to 31 days per request. Repository events include Tag Pushed, Tag Deleted and Privacy Changed; organisation events include Access token created, updated, revoked and deleted. Export to CSV with the window filter applied. An unexpected Tag Pushed on an image you deploy is the event that matters most, because it puts attacker code into your next deploy.

For ghcr.io, revoke the old personal access token on GitHub. Removing it from Railway did not kill it.

5. Cloud accounts

AWS. Event history is per Region, so search each one:

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  aws cloudtrail lookup-events --region "$r" \
    --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAEXAMPLEKEYID \
    --start-time 2026-10-20T13:00:00Z --max-results 50 \
    --query 'Events[].[EventTime,EventName,EventSource]' --output text
done

Railway's AWS guide flags RunInstances, CreateUser, CreateAccessKey, AttachUserPolicy, CreateRole and PutBucketPolicy, plus any activity in Regions you do not use.

Google Cloud. Railway's GCP guide notes that service account keys do not expire by default, and that tokens already minted from a disabled key stay valid for up to an hour. Disable the key, and disable the account itself if you need access cut immediately:

gcloud iam service-accounts keys disable KEY_ID \
  --iam-account=SA_NAME@PROJECT.iam.gserviceaccount.com --project=PROJECT
gcloud policy-intelligence query-activity \
  --activity-type=serviceAccountKeyLastAuthentication --project=PROJECT

In Logs Explorer, query logName:"cloudaudit.googleapis.com" with protoPayload.authenticationInfo.principalEmail set to the exposed account, and check for impersonation through serviceAccountDelegationInfo. Data Access logs are off by default except for BigQuery, so reads of your data may not be logged at all. Compare billing by SKU for the window; the guide names compute, GPU and Vertex AI as the common targets.

6. Databases

Railway asks every customer to remove public network access to databases. If a database had a public TCP proxy and its connection string was in a variable, assume someone may have connected. Review connection logs if your engine keeps them, rotate the password, and move clients to private networking.

-- PostgreSQL: rotate the application role's password
ALTER ROLE app_user WITH PASSWORD 'new-long-random-value';
-- Check who is connected right now
SELECT usename, client_addr, backend_start FROM pg_stat_activity WHERE usename = 'app_user';

7. Payments and signing secrets

Stripe keys, webhook signing secrets and JWT or session signing keys count too. Signing keys never appear in a usage graph, so they get skipped; rotate them and accept that existing sessions are invalidated.

Remediation: the order that avoids re-exposure

Railway explained one of its own decisions in a way every team can reuse. It waited until the attacker was out before telling users to roll credentials, "to prevent the credentials from becoming immediately compromised again". The attacker was confirmed out at 03:32 on 22 October, so rotations you did before that time should be done again. Rotations done on 21 October before the disclosure went out are the ones most likely to need a second pass.

The order:

  1. Rotate at the source. Create the new key at the provider, scoped as narrowly as the provider allows, with a spend cap and expiry where supported.
  2. Update Railway. Edit the variable, or the shared variable in Project Settings, so every reference picks it up.
  3. Redeploy. Containers read variables at start. Use Deploy Latest Commit from the command palette or railway redeploy --service NAME. Railway notes that redeploying a deployment from before the incident fails; deploy the latest commit instead.
  4. Revoke the old key. Only after the new deployment is healthy, so you do not take yourself down.
  5. Recreate automation credentials and reconnect GitHub. railway logout then railway login for the CLI, new RAILWAY_TOKEN or RAILWAY_API_TOKEN values in CI secrets, and a fresh GitHub connection. If you want your own clean point for buckets, railway bucket credentials --reset regenerates them.
  6. Write it down. Keep exported logs and a timestamped list of rotated keys. Railway's audit log retention depends on plan, so export now.

Our earlier post has a longer rotation runbook per credential type in What to do now: a rotation runbook.

What changed since the first disclosure

Topic21 Oct disclosure22 Oct preliminary report
Start of activity"Earlier this morning" (21 Oct)20 Oct 13:58 UTC
Entry pointSystems that store environment variablesBuild infrastructure first, then CI credentials, then the variables database
ObjectivePart of attacks aimed at compromising AI inference providersAI provider keys for distillation attacks
Scope"A subset of users"At most 1.85% of workspaces, about 100,000
Which keysNot statedCannot be identified individually because of envelope encryption
User accountsNot statedNot accessed
Source codeNot statedNot ruled out
VendorsUpstream providers notifiedGitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe and others named
BuildsAll pausedRestored for all tiers; builders moved to single-tenant
Attacker statusNot statedConfirmed out at 22 Oct 03:32 UTC

Open questions

  • Initial access. How did the attacker first escalate in build infrastructure? Railway withholds this for now. Until it is published, nobody outside can say whether the weakness was specific to Railway or shared with other platforms.
  • Which key names were searched for. Railway read them in the payload logs but has not published them. They would help customers prioritise.
  • Sealed variables. Were sealed values in the decrypted subset? Not stated.
  • Source code. Railway will confirm in the final report.
  • Abuse. Railway has not said whether any key was actually used. The wallet theft claim comes from a researcher quoted by The Hacker News and is unconfirmed by Railway.
  • Attribution. None offered. Railway says authorities are involved.
  • The 17:03 status entry. The report says an incident was declared on 20 October; the public incident QH8L8B3E starts on 21 October.

What would have helped

Nobody outside Railway can say what would have stopped this specific attack, because the exploit details are not public. What the report does show is how a single foothold turned into exposed customer secrets, and every stage has a general lesson.

Practices any team can use

If you store secrets on a hosting platform:

  1. Know where every secret lives. A list of secret names, owners, scope and the services that use them turns a breach day into a checklist. Without one, you are reading dashboards one at a time.
  2. Prefer short lived credentials. Workload identity federation for cloud access, OIDC from CI to cloud, and tokens that expire in minutes leave nothing durable to steal. A static key that "doesn't expire by default", in Railway's description of Google keys, is the opposite.
  3. Cap and expire AI keys. Separate keys per service and environment, spend limits and expiry dates where the provider supports them. A capped key turns abuse into a small bill and an alert.
  4. Use static outbound IPs where you can. Railway's GCP guide points out that with static egress, calls from any other address point at an attacker; with shared egress, an unfamiliar IP is only a lead.
  5. Keep databases off the public internet. Private networking means a leaked connection string is not enough on its own.
  6. Export logs before you need them. GitHub organisation logs keep 180 days, Docker Home 90, AWS CloudTrail event history 90. Send them somewhere you control.
  7. Rehearse rotation. Rotating dozens of keys across providers for the first time, while builds are paused, is slow and error prone.

If you run a platform, a CI system or anything that builds other people's code:

  1. Treat builders as hostile. Customer code runs there; Railway's move to single-tenant builders points the same way.
  2. Keep CI away from decryption authority. CI credentials here led to decrypted artifacts with more API access. Scope CI secrets to the job that needs them.
  3. Log every unwrap. If each data key decryption is a logged call to a key service with per object context, you can tell customers exactly which secrets were decrypted.
  4. Build a kill switch and a mass revoke before you need them, as Railway did by pausing builds while running services stayed up.
  5. Page on anomalies, not just alerts. Railway's first signal was elevated latency.

Where Safeguard fits

Be clear about the limits first. This incident happened inside a hosting provider's build systems, CI environment and secrets database. Safeguard does not run inside your provider, does not hold or rotate your provider keys, and cannot see whether a variable on someone else's platform was decrypted. We do not claim it would have prevented or detected this breach.

Where Safeguard does help is on your side of the line, in the code and images you build and ship:

  • Knowing what you run. SBOM and dependency scanning of repositories, container images and SBOM files (safeguard scan --dir ., safeguard scan --image <name>, safeguard scan --sbom <file>, with JSON results under ./output). The platform shows which projects contain a given component and version. After a provider breach, the follow on question is often which of your services use a particular SDK or client library to talk to that provider; that becomes a query.
  • Keeping dependencies current. Vulnerability matching against vulnerability databases, with reachability analysis to rank the code paths your application actually calls.
  • Stopping regressions. Policies and gates evaluated in the platform at build, CI and release.
  • Approved changes during a messy week. Rotation day means many small code and configuration edits: moving a hard coded client setup to read from a variable, adding a key ID to logs, switching a library to workload identity. The Safeguard CLI's assistant can make those edits and run commands with your approval, and /permissions offers Ask, Accept edits, Plan, Auto (the default) and Strict. safeguard exec runs in CI. The desktop and phone apps can send work to a computer running the CLI and approve each command.
  • Fix pull requests, off by default. Autonomous remediation opens fix pull requests for vulnerable dependencies and ships with every category set to Manual.

Zero-Day Discovery and Gold

There is no CVE in Railway's report and no known vulnerable open source package. Zero-Day Discovery looks for previously unknown flaws before a CVE exists, in an agentic mode and a deterministic scanner mode, issues findings as SGZ advisories before they reach public CVE databases, and covers seven package ecosystems. A hosting provider's internal build and CI systems are outside what it looks at, and we make no claim that it found anything related to this incident.

Gold Open Source (gold.safeguard.sh) is free CVE intelligence with CVSS, EPSS, KEV status, affected and fixed versions and a recommended safe version, plus a free read API and a CI gate GitHub Action. Gold Registry (registry.safeguard.sh) supplies hardened, Sigstore signed components and container images held to zero exploitable critical or high CVEs with SLSA L3 or higher provenance. Neither has a record tied to this incident. The habit they support is the right one after any platform breach: when you redeploy on fresh credentials, rebuild from patched, signed components rather than reusing whatever image was cached.

Common mistakes

  • Starting the search on 21 October. The report puts the first activity at 13:58 UTC on 20 October.
  • Rotating before the attacker was out. Rotations finished before 03:32 UTC on 22 October may need a second pass, by the same logic Railway used for its own keys.
  • Assuming a vendor revocation email means you were breached, or that no email means you were not. Railway says vendor resets went wider than the exposed set, and it cannot tell which values were decrypted.
  • Deleting a key before reading its usage. Railway's OpenRouter guide notes that usage fields disappear with a deleted key. Read, export, then delete.
  • Forgetting the token still works elsewhere. GitHub tokens removed from Railway's registry settings still work on GitHub until you revoke them there.
  • Treating sealed as safe. Sealed values are hidden from the UI and API but still delivered to builds and deployments. Rotate them too.

FAQ

What is Railway's preliminary incident report?

It is Railway's 22 October 2026 follow up to its 21 October security disclosure. It gives a UTC timeline of an attack from 20 October 13:58 to 22 October 03:32, says the attacker was after AI provider API keys, bounds exposure at 1.85% of workspaces and lists Railway's fixes. A final forensic report is promised.

How did the attacker get from builds to my secrets?

Railway describes the order, not the exploits: privilege escalation in build infrastructure, wider parallel escalation, a pull of encrypted secrets, then credentials for Railway's CI environment that allowed decryption of artifacts holding more API access, and finally a query of encrypted environment variables of which a subset was decrypted.

Why can't Railway tell me whether my specific key was decrypted?

The variables were envelope encrypted. Railway can see that encrypted rows were queried but says the query alone cannot identify which keys were decrypted. So it recommends rotating everything stored before 21 October and is contacting customers by risk.

What is a distillation attack?

Training a weaker model on a stronger model's outputs, done at scale against a provider's terms to copy its capabilities. Anthropic described such campaigns in February 2026. Stolen customer keys let an attacker spread that traffic across real, paying accounts that are harder to ban.

Was my source code accessed?

Railway has not ruled it out and says it will confirm in the final report. If your repositories contain anything sensitive, treat this as open.

I got a revocation email from GitHub, OpenAI or another vendor. Was my key stolen?

Not necessarily. Railway sent some vendors every key it held for their service, and says a vendor email does not imply your credential was affected. Deploy the replacement and review usage anyway.

Did attackers steal cryptocurrency?

The Hacker News quoted a researcher who said wallets with private keys stored on Railway were drained and named a receiving address. Railway's report does not mention it. If you stored wallet keys on Railway, move funds to a new wallet whose key has never been on the platform.

Is Railway safe to use now?

Railway says it has revalidated all impacted systems, rebuilt tooling from validated source, moved builders to single-tenant environments and rolled its own keys twice. The rotation steps apply either way, and so do the lessons for any platform that stores your secrets, as the Zeabur shutdown showed.

Fix it without waiting for upstream

Railway's breach is not a dependency vulnerability, and nothing below rotates a key. But a redeploy across every service is exactly the moment to make sure what you ship is current, so the same week fixes both.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

Your situationWhat to use
A fixed version existsAutonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version
You need to know today whether you are affectedGold Open Source lookup and "Am I affected?" checker, free and without login
You want vulnerable versions to stop shippingThe Gold scan GitHub Action in CI, plus Safeguard policies and gates
Upstream has no fix yet, or will not ship oneA custom Gold request; fork and patch (early access)
You want to start clean on new projectsPull hardened components and images from Gold Registry

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would have helped

These are real product sessions on other examples. None of them shows Railway, this breach or any customer's keys, and none of them rotates a credential at a provider for you.

The CLI demo shows the assistant fixing a lodash CVE: it reads the finding and advisory, asks before each command, edits package.json, runs install and tests with your approval, prints a summary and pushes nothing. For Railway, you would open the CLI in a service's repository and ask it to move a hard coded AI client setup to read its key from an environment variable, or to bump the base image before you redeploy on new credentials. Every command and edit waits for your approval.

Click through it at your own pace, or open it full screen.

The desktop app demo shows a CLI session in the conversation list, a folder choice, and Approve or Deny on each command and file edit, ending with the upgrade done and tests passing. Here you would pick the repository behind a Railway service, ask for the change, and approve each edit yourself before running railway up or pushing for an autodeploy.

Click through it at your own pace, or open it full screen.

The mobile demo shows a real Android session: work sent to a computer running the CLI, and the command approved on the phone with Allow once, Allow for turn or Deny. On a rotation day spread across many services, that lets the person on call approve each scripted step from wherever they are, without handing the assistant open ended permission.

Click through it at your own pace, or open it full screen.

To try it on your own systems, download Safeguard and read the CLI documentation. For related reading, see are environment variables on hosting platforms safe for secrets?, the LiteLLM post-mortem on agent tokens in gateway logs and October 2026 security incidents and what would have helped.

Quick check

When should your log search for the Railway incident start?

At 13:58 UTC on 20 October 2026, when Railway's report says the attack window opened. The first disclosure only said "earlier this morning" on 21 October, so searches based on it start too late.

Why does Railway recommend rotating every variable instead of naming the exposed ones?

The variables were envelope encrypted, and Railway says the attacker's query alone cannot show which keys were decrypted. It can bound the exposure at 1.85% of workspaces, but not identify individual values, so every secret stored before 21 October is treated as exposed.

Sources

Last reviewed: 23 October 2026. This is a developing story; check the linked advisories for the latest status.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.