For a repeatable PySpark environment, build the whole Python environment once, pack it into an archive, and ship that archive with the job, instead of installing packages on each node. The Spark docs describe py-files, conda, virtualenv, PEX and uv, and they differ in how much they capture.
The options in the Spark docs
The PySpark package management guide starts from a plain problem: your code and every library it uses must be available on the executors. For example, a pandas UDF needs pyarrow on each executor, or you get a ModuleNotFoundError. It lists five approaches.
| Approach | What it ships | Catch |
|---|---|---|
| Native py-files | Individual .py files, zips or eggs | Cannot add wheels, so no packages with native code |
| Conda with conda-pack | Interpreter plus all packages in a relocatable archive | You must build and store the archive |
| Virtualenv with venv-pack | Packages in an archive | The interpreter is packed as a symbolic link, so every node needs the same interpreter |
| PEX | A single executable bundle | Another artifact to build |
| uv | Environment resolved with uv | Workflow differs from the others |
The native option is fine for shipping your own modules. It is the wrong tool for third party libraries, because the docs note it does not allow wheels and therefore does not include dependencies with native code.
Why archives help reproducibility
Installing with pip on every node means every node resolves versions at its own start time. Nodes added later can differ from nodes added earlier. A packed archive fixes the question: one build, one artifact, shipped to all executors with the archives option. The conda route is the most complete, since the docs say the archive captures the interpreter and all relevant dependencies.
The archive is only as pinned as the build that made it. Build it from a lockfile, with exact versions, ideally with hashes. See pin Python dependencies in data projects and how pip hash checking mode works.
A practical routine
- Keep a lockfile for the environment in your repository.
- Build the archive in CI from the lockfile, not on a laptop.
- Store the archive with a version and a checksum, and keep the lockfile next to it.
- Reference it in the job by version, so a rerun six months later uses the same bits.
- Scan the lockfile or the unpacked environment and record the result.
- Rebuild on a schedule to pick up fixes, and treat each rebuild as a reviewed change.
One caution on the venv route
Because the venv approach packs the interpreter as a symbolic link, the docs say it requires the same interpreter on all nodes. If your cluster image changes its Python version, the archive can break or behave differently. Pin the interpreter version too, or use the conda route which carries it.
Try it yourself
Click through it at your own pace, or open it full screen.
Open the inventory and note how many packages sit beneath the ones you chose. An archive carries all of them, so all of them need to be known.
Safeguard's SBOM and dependency scanning reads lockfiles like these, and the free scan at gold.safeguard.sh/scan is a quick way to see the result.
Quick check
Can py-files ship pandas or pyarrow?
Not reliably. The Spark docs say it does not support wheels, so it cannot include dependencies with native code. Use a packed environment instead.
Does packing an environment pin versions by itself?
No. It freezes whatever was installed at build time. Pin versions in the lockfile you build from, so the build itself is repeatable.
Sources
- Python Package Management, PySpark documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
