Several comfortable beliefs about data pipeline tools do not match their documentation. Hidden values can still show, installs can drift, and workers may or may not touch the database. Here are six myths set against what the official docs say, checked on 19 October 2026.
Myth 1: "Environment variables are never shown in the UI"
Fact: it depends how you read them. Dagster's docs say that with os.getenv the value is retrieved when the code location loads and will be visible in the UI, while EnvVar resolves at run time and is not visible. See Dagster EnvVar or os.getenv.
Myth 2: "Spark hides all secrets in its logs"
Fact: Spark redacts values whose property key or value matches spark.redaction.regex, which by default looks for secret, password, token and access key. A property with a different name can show. See what spark.redaction.regex hides.
Myth 3: "Our jobs install the same packages every time"
Fact: AWS Glue's docs say that with the additional Python modules parameter alone, pip resolves dependencies from PyPI on every run, so a new release can change what the job installs. Spark's packages option similarly resolves Maven coordinates and their transitive dependencies at submit time. See Glue and pip drift.
Myth 4: "Airflow workers can read the metadata database"
Fact, for Airflow 3: the security model says worker task code does not receive database credentials and communicates through the Execution API. But DAG parsing and trigger code run in components that do have database access, and tasks are not separated per DAG. See Airflow 3 task database access.
Myth 5: "A database user for CDC must be a superuser"
Fact: Debezium's PostgreSQL docs say a superuser works but potentially exposes your data to unauthorized access, and recommend a dedicated replication user with minimum privileges. See Debezium user privileges.
Myth 6: "Connector settings are always locked down by the platform"
Fact: Kafka Connect's default connector client override policy is All, meaning connector configs can override all client properties. None and Principal are tighter options. See the override policy explained.
What the myths have in common
Each belief treats a convenience feature as a control. A hiding feature hides a value from one screen. A default policy is a starting point, not a decision. A pinned top level package leaves the layers below free to move. The fix is to read what the feature actually claims, then test it in a non-production setup.
A short test plan:
- Put a dummy secret in each tool's config and see where it appears.
- Run the same job twice, a week apart, and compare installed packages.
- Check which components in your orchestrator hold database credentials.
- Read the default for every policy setting and write down whether you chose it.
Where Safeguard fits
For the code and packages around your pipelines, Safeguard offers SBOM and dependency scanning and policy gates. The free scan is at gold.safeguard.sh/scan.
Quick check
Where should I start testing?
With the dummy secret test and the repeat install comparison. Both take under an hour and show quickly which of these beliefs holds in your setup.
Do these facts apply to every version?
No. Defaults and features change by release. Check the docs for the version you run before relying on any of them.
Sources
- Using environment variables and secrets, Dagster documentation, fetched 19 October 2026.
- Spark Configuration, Apache Spark documentation, fetched 19 October 2026.
- Using Python libraries with AWS Glue, AWS documentation, fetched 19 October 2026.
- Submitting Applications, Apache Spark documentation, fetched 19 October 2026.
- Airflow security model, Apache Airflow documentation, fetched 19 October 2026.
- Debezium connector for PostgreSQL, Debezium documentation, fetched 19 October 2026.
- Kafka Connect configs, Apache Kafka documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
