No single mechanism hides a pipeline secret everywhere. Each tool's approach keeps the value out of one place, such as a config file, the UI or a stored connector config, and leaves other places open. Compare them by what each prevents, then add the controls they leave out.
The approaches side by side
| Mechanism | Keeps the secret out of | Still exposed to |
|---|---|---|
| Airflow config _cmd and _secret suffixes | Plain text airflow.cfg | Anyone running code as the Airflow user; a stale plain variable can override it |
| Dagster EnvVar | The UI, since it resolves at run time | Code that prints it; the deployment environment |
| Prefect blocks with SecretStr fields | UI display by default; stored with extra encryption | Flow code that loads the block |
| Kafka Connect ConfigProvider | Stored connector configs, logs and REST responses | The worker process, and the secret store itself |
| Spark redaction regex | UI and logs, for names or values that match the pattern | Names that do not match; code that logs values |
Each row comes from that project's own documentation. They solve different problems, so they are not interchangeable, and none of them stops an authorized pipeline from using the secret as intended.
Four questions to ask of any mechanism
- Where is the value stored, and is it encrypted there? A config file, a database, a vault.
- Who can read it through the UI or API? Many tools show config by default.
- Does it appear in logs, event histories or exports? Check for yourself, as with the Spark test described in what spark.redaction.regex hides.
- Who can write code that loads it? That person can use it, whatever the UI shows.
The fourth question is the one that decides risk. Display controls limit casual exposure. The real boundary is code review and deployment rights.
A layered pattern
- Store secrets in one managed store, not in each tool.
- Use short lived credentials where the target system supports them.
- Give each pipeline its own identity with only the access it needs, so a leak affects one pipeline.
- Keep secrets out of code, config files and command lines.
- Rotate on a schedule, and immediately when someone with access leaves.
- Test what the UI and logs show, using a harmless dummy secret.
Checking an existing estate
Walk through each orchestrator and connector you run:
- List the secrets it holds and who created them.
- Check which mechanism protects each one, using the table above as a prompt.
- Check the UI, logs and API output for clear text values.
- Find secrets older than your rotation policy and rotate them.
- Remove shared credentials, replacing them with one per pipeline.
For detail on specific tools, see Airflow secrets backend lookup order, dbt secrets in environment variables and logs and ETL pipeline secrets: where credentials leak.
Safeguard is not a secrets store, and this post does not describe scanning for secrets. It does cover dependency scanning, SBOMs and policy gates for the code around your pipelines; try the free scan at gold.safeguard.sh/scan.
Quick check
Is hiding a secret in the UI enough?
No. It limits casual exposure only. Code that can load the secret can still use or print it, so control who can deploy pipeline code.
Why one credential per pipeline?
A leak or misuse then affects one pipeline, and you can rotate or revoke without breaking the others.
Sources
- Setting configuration options, Apache Airflow documentation, fetched 19 October 2026.
- Using environment variables and secrets, Dagster documentation, fetched 19 October 2026.
- Blocks, Prefect documentation, fetched 19 October 2026.
- Kafka Connect security, Confluent documentation, fetched 19 October 2026.
- Spark Configuration, Apache Spark documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
