For a single file pipeline, a curated requirements file is the simplest way to give Beam workers their packages. For anything more complex, the Beam docs suggest packaging the pipeline and installing dependencies in a custom container. Either way, the pinned versions you write are what makes reruns repeatable.
What the Beam docs describe
The Apache Beam guide to Python pipeline dependencies says that if your pipeline uses public packages from PyPI, you must make them available remotely on the workers. For a pipeline that is a single Python file or notebook, the most direct route is a requirements file. For more complex cases, define the pipeline as a package and consider installing dependencies in a custom container.
To create the file, the docs suggest running pip freeze into a requirements file, and then editing it to delete everything your pipeline does not need. They note that pip freeze lists all packages installed on your machine regardless of where they came from. As an alternative they point to a tool such as pip-tools, where you list only top-level dependencies in an input file and compile the full set.
How staging works
When you supply the requirements file option, Beam downloads the listed packages into a local requirements cache at submission and stages that cache to the runner. At runtime, workers can install from the cache, possibly without a connection to PyPI. So the resolution happens on the machine that submits the job, which makes that machine part of your supply chain.
Comparing the approaches
| Approach | Strength | Weak point |
|---|---|---|
| Raw pip freeze output | Fast | Captures every package on your laptop, including unrelated ones |
| Curated requirements file with exact versions | Small, readable | Transitive dependencies need pinning too |
| Compiled file from a top-level input | Repeatable and complete | Needs a compile step in CI |
| Custom container | One fixed environment for all workers | An image to build, scan and store |
The freeze output deserves a second look. A laptop environment can hold tools, test libraries and packages from earlier experiments. Staging all of that onto workers widens what runs against your data. Trim it, and build the file in CI rather than on a developer machine.
A sound routine
- Keep a short input file of direct dependencies.
- Compile it to a fully pinned file in CI, with hashes if you can.
- For production, build a container from that file and reference the image by digest.
- Scan the image or the pinned file and keep the result with the release.
- Pin the Beam SDK version in the same file, since it is also a dependency.
- Rebuild on a schedule, and review the diff.
See pin Python dependencies in data projects and container image digests vs tags: why pinning matters.
Try it yourself
Click through it at your own pace, or open it full screen.
Watch how an image is broken into layers and packages. A worker container for a pipeline is built from layers like these.
Safeguard's SBOM and dependency scanning, including container images, covers this. A free scan is at gold.safeguard.sh/scan.
Quick check
Is pip freeze output safe to use as is?
The Beam docs say it lists every installed package regardless of origin, and tell you to remove what the pipeline does not need.
When does a custom container make sense?
For more complex pipelines, as the Beam docs suggest. It gives every worker the same fixed environment that you can scan and store.
Sources
- Managing Python pipeline dependencies, Apache Beam documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
