Rotating a leaked CI secret without downtime is a four-phase operation: issue a new credential alongside the old one, cut every consumer over to the new one, verify from usage logs that the old one has gone quiet, and only then revoke it — with one deliberate exception, because active exploitation collapses the sequence to "revoke now and eat the outage." Teams cause their own incidents here in two symmetric ways: revoking instantly and breaking every pipeline that still holds the old value, or delaying revocation for days while the leaked credential stays live. The dual-credential pattern avoids both. Here it is, phase by phase.
Phase 0: triage before you touch anything
Ten minutes of scoping changes the whole plan. Establish three facts:
- What can this credential do? An
npmpublish token and a read-only artifact-pull token are different emergencies. - Where did it leak? A secret printed once in a CI log expires with log retention; a secret committed to a public repo was cloned by scanners within minutes — GitHub's own data and every honeytoken study agree exploitation of public leaks starts in under five minutes. Public leak means assume compromise.
- Is it being used maliciously right now? Check the provider's audit trail — CloudTrail for AWS keys, the npm token audit log,
gcloud logging readfor service account keys. Active abuse means skip to revocation and accept the downtime; an outage beats an attacker with a live credential.
Record timestamps as you go. If this turns into a reportable incident, the timeline you write in the first hour is the one your auditors and customers will read.
Phase 1: issue the new credential alongside the old
Almost every system you care about supports two live credentials, precisely to enable this:
- AWS: an IAM user can hold two access keys —
aws iam create-access-key --user-name ci-deployerwhile the old key still works. - Databases: create a second user with identical grants (
ci_app_2026a), or use paired users you alternate between. - API providers: GitHub, npm, Stripe, Datadog all allow multiple active tokens; mint the replacement with the minimum scopes the pipeline actually uses — rotation day is the best scope-tightening opportunity you will ever get.
Store the new value in your secret manager as a new version, not a manual edit in five CI configs. If your secrets live as flat CI variables copied across projects, this incident is the argument for centralizing: GitLab group-level variables, GitHub organization secrets, or Vault/AWS Secrets Manager with CI reading at runtime. Our Secrets Manager vs Parameter Store comparison covers the AWS-side tradeoffs.
Phase 2: cut consumers over, in order of blast radius
Update the secret store, then force consumers to pick it up. The failure mode to avoid is the half-migrated state living longer than planned, so inventory consumers first — pipelines, scheduled jobs, the odd developer laptop, and the service that "temporarily" got the CI credential in 2024. Grep for the variable name across your org, not just the repo that leaked it.
For runtime services, roll deployments gradually and watch auth-failure metrics between batches. For CI, trigger a representative pipeline per project and confirm green. Nothing here should be novel — it is an ordinary config rollout, which is the point of doing it under dual-credential cover: both values work throughout, so a straggler pipeline is a cleanup item, not an outage.
Phase 3: verify the old credential is quiet, then revoke
Do not revoke on faith. Revoke on telemetry:
# AWS: when was the old key last used, and for what service?
aws iam get-access-key-last-used --access-key-id AKIA...OLD
# CloudTrail: anything still authenticating with it?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...OLD \
--start-time 2026-03-12T00:00:00Z
Wait at least one full cycle of your slowest consumer — if a nightly job uses the credential, that means 24 hours — then deactivate before deleting (aws iam update-access-key --status Inactive). Deactivation is reversible; if something obscure breaks an hour later, you flip it back, fix the consumer, and try again. Delete after a quiet day or two. One caveat: for a publicly leaked credential, this patience applies only if Phase 0 showed no abuse, and even then, deactivate within hours, not days — "no abuse yet" has a short shelf life.
Phase 4: assume the credential was used, and check
Rotation ends the exposure; it does not undo what happened during it. Review the audit trail for the full leak window, not just today: resources created, tokens minted (an attacker's favorite move is using the leaked credential to create durable access that survives your rotation), packages published, workflow files modified. The 2023 CircleCI incident forced thousands of organizations through exactly this exercise, and the ones who struggled were those who could not answer "what did this token touch" — the write-up in our CircleCI credential rotation retrospective is worth reading before you need it.
Make the next one boring
Three structural fixes, in payoff order. First, replace static CI credentials with OIDC federation wherever the provider supports it — GitHub Actions and GitLab CI can both exchange short-lived identity tokens for cloud credentials, and a credential that expires in fifteen minutes barely merits a rotation runbook. Second, put detection in the path: gitleaks or TruffleHog as a pre-receive hook and a CI job, so leaks surface in the PR, not in a stranger's disclosure email. Third, drill the rotation per credential class once a quarter. A rotation you have rehearsed is a chore; one you improvise at 2 a.m. is an incident.
Frequently asked questions
When should I skip the graceful sequence and revoke immediately?
When audit logs show active malicious use, or when the credential grants write access to something irreversible — package publishing, code signing, production data deletion. In those cases the outage is the cheaper outcome, every time.
How long should the old and new credentials overlap?
As short as your slowest consumer allows: typically a few hours for pipeline-only credentials, up to a day or two when nightly or weekly jobs are involved. Set an explicit deadline at the start — overlap windows without owners become permanent.
Does rotating the secret end the incident?
No. Rotation closes the door; you still have to check whether anyone walked through it. Audit the leak window for persistence — new tokens, new users, modified workflows — because attackers who catch a credential immediately convert it into access that survives rotation.
What actually prevents CI secret leaks rather than just responding to them?
Short-lived OIDC-federated credentials instead of static keys, masked and scoped CI variables, and secret scanning in the PR path. Static long-lived secrets in CI are the root cause; everything else is mitigation.