To rebuild a data pipeline environment as it was, pin every layer: the Python or JVM runtime, each package including transitive ones, OS packages, the base image by digest, and the orchestrator and its providers. A pin at one layer does nothing if another layer is free to move.
Definition
A reproducible environment is one you can rebuild later and get the same software. That is related to, but not the same as, a reproducible build, where the output is bit for bit identical; see hermetic vs reproducible build. For pipelines the practical target is smaller: the same package versions, so a rerun means the same code touched the data.
The layers
| Layer | What drifts | How to pin |
|---|---|---|
| Interpreter or JVM | Minor upgrades in a base image | Fixed version in the image or environment file |
| Python packages | Newer releases, new transitive pulls | Lockfile with exact versions and hashes |
| JVM packages | Maven resolution at submit time | Exact coordinates, or a built jar |
| OS packages | Package manager updates | Version in the install command |
| Base image | A tag that moves | Digest |
| Orchestrator and providers | Separate release trains | Constraints file, pinned image |
| Pipeline code | A branch that moves | Tag or commit |
Several rows have a documented example. Spark's submitting applications guide says transitive dependencies are resolved from the coordinates you give. AWS Glue's docs say that pip resolves from PyPI on every run when you use only a module list. Databricks advises pinning OS package versions in init scripts for reproducible clusters. Beam suggests a curated or compiled requirements file, or a custom container.
A build order that works
- Choose the interpreter version and write it down.
- Write a short file of direct dependencies.
- Compile it to a fully pinned lockfile in CI.
- Install with hash checking, so a changed file fails the build.
- Build the image from a base referenced by digest.
- Install OS packages with explicit versions.
- Tag the resulting image and keep the lockfile with it.
- Run the same scan on the lockfile and the image.
What pinning does not do
Pinning stops surprise changes. It also stops surprise fixes. A pinned environment that nobody rebuilds drifts into vulnerability. So pair every pin with a schedule: rebuild, test, and review the diff, on a cadence you can defend. A pin is a decision, not a freeze.
Also record what actually ran. Keep the resolved package list per job run, so that when a new advisory lands you can search runs, not just repositories.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through the inventory and count how many packages sit below the ones you declared. Those are the pins that are easy to miss.
Safeguard's SBOM and dependency scanning builds this inventory; try the free scan at gold.safeguard.sh/scan.
Quick check
Is a lockfile enough?
It covers Python or JVM packages only. The interpreter, OS packages and base image can still change, so pin those too.
How often should I rebuild a pinned environment?
On a schedule you choose and can defend, plus when a relevant advisory lands. Pinning without rebuilding preserves known vulnerabilities.
Sources
- Submitting Applications, Apache Spark documentation, fetched 19 October 2026.
- Using Python libraries with AWS Glue, AWS documentation, fetched 19 October 2026.
- Init scripts, Databricks documentation, fetched 19 October 2026.
- Managing Python pipeline dependencies, Apache Beam documentation, fetched 19 October 2026.
- Prefect YAML deployment guide, Prefect documentation, fetched 19 October 2026.
- Python Package Management, PySpark documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
