In 2022, GitGuardian reported detecting over 10 million secrets exposed in public GitHub commits. That number only accounts for public repositories. The scale of credential exposure in private repositories, CI/CD systems, container registries, and artifact stores is unknowable but almost certainly larger.
Hardcoded credentials are not a new problem. They are a persistent one. Despite widespread awareness, secrets keep ending up where they should not be. The reasons are structural: developers need credentials to build and test software, the path of least resistance is putting them directly in code or configuration, and the consequences of a leak are not immediately visible to the person who committed it.
Fixing this requires more than awareness. It requires infrastructure, tooling, and workflow changes that make the secure path also the easy path.
Where Secrets Leak
Understanding the attack surface is the first step. Credentials escape through more channels than most teams realize:
Source Code Repositories
The classic leak. A developer hardcodes an AWS access key in a configuration file, commits it, and pushes. Even if the secret is removed in a subsequent commit, it persists in git history forever unless the history is rewritten.
# This gets committed more often than anyone wants to admit
AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
GitHub's secret scanning catches known credential formats in public repositories, but custom API tokens, internal service passwords, and non-standard credential formats slip through pattern-based detection.
Container Images
Docker images are layered. Every COPY and ENV instruction creates a layer that persists in the image even if a subsequent layer deletes the file. A common mistake:
# Secret is permanently embedded in image layer
COPY .env /app/.env
RUN npm install
RUN rm /app/.env # This does NOT remove it from the previous layer
Anyone with access to the image can extract the .env file from the earlier layer using docker history and docker save. Published images on Docker Hub and private registries frequently contain embedded credentials.
CI/CD Pipeline Logs
Build systems log extensively. When a deployment script echoes environment variables for debugging, or a test framework prints configuration details on failure, secrets appear in build logs. These logs are often retained for weeks or months and accessible to broader teams than production credentials should be.
# Dangerous: secret will appear in build log
- name: Deploy
run: |
echo "Deploying with key: $API_KEY"
curl -H "Authorization: Bearer $API_KEY" https://api.example.com/deploy
Most CI systems mask known secret variables in logs, but this masking is pattern-based and imperfect. A secret passed through base64 encoding, string concatenation, or any transformation will not be masked.
Dependency Configuration
Package managers sometimes require authentication tokens for private registries. These tokens end up in .npmrc, pip.conf, settings.xml, and similar configuration files that get committed alongside source code:
# .npmrc committed to repository
//npm.pkg.github.com/:_authToken=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
@company:registry=https://npm.pkg.github.com
Build Artifacts and Bundles
Frontend JavaScript bundles occasionally embed API keys that were meant to be server-side only. Source maps, if published, can reveal credentials that were present in the original source. Compiled binaries can contain string literals with embedded credentials that are trivially extractable.
The Secrets Management Stack
A proper secrets management implementation has multiple layers, each addressing a different part of the problem.
Layer 1: Secret Storage (Vault)
Centralized secret storage is the foundation. HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Secret Manager all provide the same core capability: storing secrets in an encrypted, access-controlled, auditable system.
The critical features are:
Dynamic secrets — generating short-lived credentials on demand rather than storing long-lived ones. Vault can generate database credentials that are valid for 1 hour and automatically revoked. An attacker who steals a dynamic credential has a narrow exploitation window.
Lease-based access — every secret retrieval has an expiration time. Applications must renew their leases or re-authenticate. This limits the damage from a single credential theft.
Audit logging — every secret access is logged with the identity of the requester, the secret accessed, and the timestamp. When a breach occurs, the audit log shows exactly what was exposed.
For most organizations, the choice between Vault and cloud-native solutions depends on infrastructure:
# AWS Secrets Manager - if you're AWS-native
aws secretsmanager get-secret-value --secret-id production/database/credentials
# HashiCorp Vault - if you're multi-cloud or on-prem
vault kv get -field=password secret/production/database
Layer 2: Secret Injection
Applications need credentials at runtime. The injection method determines your security posture:
Environment variables are the most common approach and work across every language and framework. They are better than hardcoding but have limitations — they are visible to any process running as the same user, they appear in /proc/<pid>/environ on Linux, and they tend to get logged accidentally.
Mounted files (via Kubernetes Secrets, Vault Agent, or similar) provide credentials as files in a tmpfs mount. The application reads the file at startup. This avoids environment variable leakage but requires the application to handle file-based configuration.
Direct API calls from the application to the secrets manager provide the tightest control. The application authenticates to Vault (or equivalent) using its workload identity and retrieves only the secrets it needs. This approach supports dynamic secrets and lease renewal:
import hvac
client = hvac.Client(url='https://vault.internal:8200')
# Authenticate using Kubernetes service account
client.auth.kubernetes.login(role='web-app', jwt=sa_token)
# Retrieve database credentials - these are dynamically generated
creds = client.secrets.database.generate_credentials('web-app-db')
db_username = creds['data']['username']
db_password = creds['data']['password']
# Credentials auto-expire after the configured TTL
Layer 3: Pre-Commit Detection
Catching secrets before they reach the repository is the most cost-effective control. Once a secret is in git history, remediation requires rotating the credential and potentially rewriting repository history.
git-secrets (from AWS Labs) checks commits against known credential patterns:
git secrets --install
git secrets --register-aws
# Now git commit will fail if AWS credentials are detected
Gitleaks provides broader detection with a configurable ruleset:
# .gitleaks.toml
[rules]
description = "Generic API Key"
regex = '''(?i)(api[_-]?key|apikey)[\s]*[=:]+[\s]*['"]?([a-zA-Z0-9]{32,})['"]?'''
tags = ["api-key"]
TruffleHog scans repository history, not just the current commit. This catches secrets that were committed and then "removed" (but still present in history).
Pre-commit hooks run locally on developer machines. They are not a substitute for server-side scanning because developers can bypass hooks with --no-verify. Both layers are necessary.
Layer 4: CI/CD Pipeline Scanning
Server-side scanning catches what pre-commit hooks miss. Every pull request should be scanned for secrets before merge:
# GitHub Actions secret scanning
- name: Scan for secrets
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
This scan should block the PR if secrets are detected. A soft warning that developers can ignore is functionally equivalent to no scanning at all.
Layer 5: Runtime Monitoring
Secrets leak through channels other than source code. Runtime monitoring covers:
- Container image scanning for embedded credentials in image layers
- Log monitoring for credential patterns appearing in application and infrastructure logs
- Network monitoring for credentials transmitted in plain text or to unexpected destinations
- Cloud API monitoring for credential usage from unusual IP addresses or geographies
Credential Rotation
Detection is necessary but not sufficient. Every credential should have a rotation schedule, and the rotation process should be automated:
Short-lived credentials (minutes to hours): OIDC tokens, Vault dynamic secrets, STS temporary credentials. These rotate automatically by design.
Medium-lived credentials (days to weeks): API tokens, service account keys. Automate rotation using your secrets manager's rotation feature.
Long-lived credentials (months or more): Root account passwords, encryption keys. These should be extremely rare in a mature secrets management program. Every long-lived credential represents a risk that compounds over time.
The rotation process must be tested. An automated rotation that breaks production because the application was not designed to handle credential changes is worse than manual rotation. Applications should be built to gracefully handle credential refresh — retry with a new credential on authentication failure, watch for file changes on mounted secret files, or implement lease renewal for Vault-sourced credentials.
Incident Response for Leaked Secrets
When a credential leak is detected, the response must be immediate:
- Rotate the credential. Do this first, before investigation. A leaked credential is compromised until proven otherwise, and the cost of an unnecessary rotation is negligible compared to the cost of exploitation.
- Determine the exposure window. When was the credential first exposed? When was it rotated? This window defines the scope of potential compromise.
- Audit usage during the exposure window. Review access logs for the compromised credential. Look for access from unexpected IPs, unusual API call patterns, or data access outside normal patterns.
- Identify the root cause. How did the credential leak? Hardcoded in source? Logged in CI? Embedded in a container image? The root cause determines which preventive control needs strengthening.
- Verify the fix. Confirm that the new credential is not exposed through the same channel and that the preventive control is in place to catch future occurrences.
How Safeguard.sh Helps
Credential leaks are a supply chain problem. When a secret is embedded in a dependency, a container image, or a build artifact, it moves through your software supply chain and potentially into customer environments. Traditional secret scanning catches credentials in source code but misses them in compiled artifacts, container layers, and distributed packages.
Safeguard.sh provides supply chain visibility that complements dedicated secret scanning tools. By maintaining comprehensive SBOMs and continuously monitoring your software artifacts, Safeguard.sh helps identify the components and build processes where credential exposure is most likely. When a leaked credential is discovered, Safeguard.sh's dependency tracking shows exactly which builds, images, and deployments may contain the compromised credential, enabling targeted rotation and remediation instead of organization-wide fire drills.
Combined with Safeguard.sh's policy enforcement capabilities, teams can codify secrets management requirements — ensuring that container images are scanned before deployment, that build artifacts meet security standards, and that the software supply chain does not become a vector for credential exposure.