Automating secret rotation in Google Cloud Secret Manager

Secrets don't stay secret forever—and they shouldn't. API keys, database passwords, and service account tokens that never rotate are a standing invitation to attackers: steal one once, and it stays valid indefinitely. Yet in most organizations, GCP Secret Manager rotation is treated as a checkbox exercise—configured once, rarely audited, and often decoupled from the actual credential change it's supposed to trigger. The result is stale secrets, silent rotation failures, and services that break the moment someone finally does rotate manually.

This guide walks through building real automatic secret rotation GCP workflows: wiring Secret Manager's native rotation scheduling to a Cloud Function that generates new values, updates the source system, and publishes a fresh secret version—without downtime. By the end, you'll have a repeatable pattern to rotate database credentials GCP-wide, verify it's working, and extend it to any secret type in your environment.

Step 1: Understand How GCP Secret Manager Rotation Works

Secret Manager doesn't rotate values for you out of the box. What it provides natively is a rotation schedule—a timer that fires a Pub/Sub notification at a configured interval—and Secret Manager versioning, which lets you add a new value to a secret without invalidating the old one until you're ready.

The actual rotation logic (generating a new credential, updating the downstream system, adding the new version, disabling the old one) is your responsibility, typically implemented as a Cloud Function or Cloud Run job triggered by that Pub/Sub message. This split matters: it means GCP Secret Manager rotation is really an event framework, not a magic button, and your automation needs to handle three things every cycle—generate, propagate, and retire.

Before building anything, inventory what needs rotation: database passwords, third-party API keys, signing keys, and service account keys. Each has a different rotation mechanism on the source side, so plan for a small library of rotation handlers rather than one generic script.

Step 2: Enable Rotation Scheduling on a Secret

Start by creating (or updating) a secret with a rotation policy and a Pub/Sub topic that will receive rotation events:

gcloud pubsub topics create secret-rotation-notices

gcloud secrets create db-app-password \
  --replication-policy="automatic" \
  --rotation-period="2592000s" \
  --next-rotation-time="2026-08-05T00:00:00Z" \
  --topic="projects/PROJECT_ID/topics/secret-rotation-notices"

--rotation-period is expressed in seconds (2,592,000s ≈ 30 days). Every time the rotation timer elapses, Secret Manager publishes a message to the topic containing the secret's resource name—it does not include the secret value, so this is safe to log and monitor. You can inspect or adjust an existing secret's policy at any time:

gcloud secrets update db-app-password \
  --rotation-period="2592000s" \
  --next-rotation-time="2026-08-05T00:00:00Z"

Step 3: Build a Rotation Handler with Cloud Functions

Create a Cloud Function subscribed to the Pub/Sub topic. Its job is to generate the new credential value, write it into the source system (a database, an external API, an IAM key store), and add it as a new Secret Manager version.

import functions_framework
from google.cloud import secretmanager
import secrets, string

client = secretmanager.SecretManagerServiceClient()

def generate_password(length=32):
    alphabet = string.ascii_letters + string.digits + "!@#%^&*"
    return "".join(secrets.choice(alphabet) for _ in range(length))

@functions_framework.cloud_event
def rotate_secret(cloud_event):
    secret_name = cloud_event.data["message"]["attributes"]["secretId"]
    parent = f"projects/PROJECT_ID/secrets/{secret_name}"

    new_password = generate_password()

    # 1. Update the credential on the source system first
    update_database_password(secret_name, new_password)

    # 2. Add the new value as a fresh Secret Manager version
    response = client.add_secret_version(
        request={
            "parent": parent,
            "payload": {"data": new_password.encode("UTF-8")},
        }
    )
    print(f"Added {response.name}")

Deploy it with an event trigger bound to the topic:

gcloud functions deploy rotate-db-secret \
  --gen2 \
  --runtime=python312 \
  --region=us-central1 \
  --source=. \
  --entry-point=rotate_secret \
  --trigger-topic=secret-rotation-notices \
  --service-account=secret-rotator@PROJECT_ID.iam.gserviceaccount.com

Grant that service account roles/secretmanager.secretVersionAdder on the specific secret rather than project-wide secretmanager.admin—least privilege matters most on the identity that's allowed to mint new secret material.

Step 4: Wire the Downstream System Before You Cut Over

The order of operations in Step 3 is deliberate: update the source system first, then publish the new secret version. If you publish the new version before the database (or API) actually accepts the new password, any service that reloads the secret mid-rotation will start failing authentication immediately.

For Cloud SQL, update the database user directly as part of the handler:

gcloud sql users set-password app_user \
  --instance=prod-primary \
  --password="$NEW_PASSWORD"

For third-party APIs, call their credential-rotation endpoint (most support issuing a new key before revoking the old one) so both values are valid briefly. That overlap window is what makes zero-downtime rotation possible—it gives already-running services time to pick up the new version on their normal refresh cycle instead of failing outright.

Step 5: Rotate Database Credentials Without Breaking Live Connections

To rotate database credentials GCP applications depend on without an outage, keep both the old and new secret versions active simultaneously for a grace period:

# Add new version (done by the Cloud Function)
# ... then, after confirming services have picked it up:

gcloud secrets versions disable 3 --secret="db-app-password"

Disable, don't destroy, the previous version initially—disable blocks access but keeps it recoverable if a laggard service is still using it, whereas destroy deletes the value permanently. A sensible pattern is: disable the previous version 24–48 hours after the new one is added, then destroy it 7 days later once you've confirmed nothing references it via audit logs.

Applications should always fetch the latest alias rather than pinning a version number:

gcloud secrets versions access latest --secret="db-app-password"

This is the piece teams most often get wrong: hardcoding a specific version number in application config silently defeats rotation, because the app keeps requesting a version that will eventually be disabled.

Step 6: Deploy and Test the Full Cycle End-to-End

Force a rotation manually to validate the pipeline before waiting for the real schedule:

gcloud pubsub topics publish secret-rotation-notices \
  --message='{}' \
  --attribute="secretId=db-app-password,eventType=ROTATE"

gcloud secrets versions list db-app-password

Confirm three things in order: a new version appears, the application can still authenticate using latest, and the old credential is rejected by the source system once its version is disabled. Run this dry run in staging first—rotating a production database password with an untested handler is how a routine maintenance task turns into an incident.

Troubleshooting and Verification

Rotation event never fires the function. Check that the Pub/Sub subscription exists and the Cloud Function's trigger is actually bound to the correct topic:

gcloud pubsub topics list-subscriptions secret-rotation-notices
gcloud functions describe rotate-db-secret --gen2 --region=us-central1

Function runs but the new version never appears. Usually a permissions gap—verify the function's service account has secretmanager.versions.add on the target secret, and check Cloud Logging for PermissionDenied errors:

gcloud logging read \
  'resource.type="cloud_function" AND resource.labels.function_name="rotate-db-secret"' \
  --limit=20 --format=json

Services fail authentication right after rotation. This almost always means the new credential wasn't fully propagated to the source system before the new secret version went live, or a service is caching the old value longer than the overlap window. Extend the disable delay and confirm your app reloads secrets rather than caching them for the process lifetime.

Rotation schedule silently stops. Rotation timers reset each time a version is added programmatically only if your handler also updates next-rotation-time; otherwise Secret Manager recalculates it automatically from rotation-period. Confirm with:

gcloud secrets describe db-app-password --format="value(rotation)"

Audit the whole history. Use Cloud Audit Logs to confirm every AddSecretVersion and DisableSecretVersion call correlates with an expected rotation event, not an unexpected manual change:

gcloud logging read \
  'protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion"' \
  --limit=50

How Safeguard Helps

Automating GCP Secret Manager rotation solves the mechanics of generating and propagating new credentials, but it doesn't answer a harder question: are your rotation policies actually covering everything they should, and is anything relying on a secret that's overdue? Safeguard continuously inventories secrets and credentials across your cloud and CI/CD footprint, flags secrets without an active rotation policy, and correlates Secret Manager versioning events with real usage so you can see exactly which services still reference a version you're about to disable.

Safeguard also extends visibility upstream of Secret Manager itself—catching hardcoded credentials in source repos, build logs, and container images that would otherwise sit outside your rotation pipeline entirely, no matter how well-automated it is. Combined with policy checks that flag secrets pinned to specific versions instead of latest, teams get an early warning before a rotation cutover breaks something, rather than finding out from a production incident. For supply chain security teams, that closes the loop between "we automated rotation" and "we can prove nothing depends on a stale secret."