When you pass Maven coordinates to the packages option of spark-submit, Spark resolves them, and all their transitive dependencies, when the job is submitted. If you leave the version loose or point at extra repositories, what runs can change between two submissions of the same code.
What the docs say it does
The Spark submitting applications guide says users can include other dependencies by supplying a comma separated list of Maven coordinates with the packages option, and that all transitive dependencies are handled. Extra repositories, or resolvers in SBT, can be added with a repositories option.
That is convenient, and it is also a download from a repository at the moment your job starts. You are not reviewing a lockfile. You are trusting that resolution produces the same answer it did last week.
Three ways this goes wrong
A moving version. A coordinate has three parts: group, artifact and version. If your tooling lets the version float to a newer release, a new upstream publish changes your job without any change in your repository. Write the full version every time.
Transitive drift. Even with your direct coordinate fixed, the dependencies underneath are resolved at that time. A direct dependency that declares a range, or a repository that gains a newer matching artifact, can shift what you get.
Credentials in the URI. The same docs note that credentials for password protected repositories can in some cases be supplied in the repository URI, in the form of user, password and host, and they add: be careful when supplying credentials this way. Anything in a command line is likely to land in shell history, process listings, CI logs and the Spark UI.
A safer pattern
| Approach | Reproducible | Review point |
|---|---|---|
| Packages option with exact versions | Mostly, for direct coordinates | Version in the submit command |
| Build a single application jar with dependencies | Yes, once built | The build and its lockfile |
| Pre-staged jars from an internal repository | Yes | Intake into the repository |
For anything that runs on a schedule or touches production data, prefer building the application jar in CI, with dependency versions locked by the build tool, and storing that artifact. The cluster then runs a file you scanned and recorded, not a fresh resolution.
If you keep the packages option for notebooks and experiments, treat those as a separate zone: no production credentials, no production data.
What to record
Whichever approach you use, write down, per job: the exact coordinates including versions, which repositories were allowed, and the resolved dependency list from a run. That last item is what a scanner or an auditor can actually use. The resolved list is your bill of materials for the job. If you cannot produce it, you cannot answer "does this job include the vulnerable library we just read about?"
See also how to inventory every dependency in a data platform and Apache Spark security settings to check.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through the inventory and look at how transitive packages appear that no one listed directly. That is the layer the packages option resolves for you.
Safeguard's SBOM and dependency scanning can build that inventory from your repositories, and the free scan at gold.safeguard.sh/scan is a way to try it.
Quick check
Does pinning the direct coordinate pin everything?
No. It fixes the top level artifact. Transitive dependencies are still resolved at submit time, so record the resolved list or build a jar with locked versions.
Is it safe to put repository credentials in the URI?
The Spark docs advise being careful. Command lines are easy to leak into logs and listings, so prefer a mechanism that keeps the secret out of the command.
Sources
- Submitting Applications, Apache Spark documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
