A shelf is safe only if you trust whoever wrote the file. The Python documentation warns that because shelve is backed by pickle, it is insecure to load a shelf from an untrusted source, and that loading one can execute arbitrary code. For state shared across jobs or teams, use a plain data format.
What shelve is
The standard library describes a shelf as a persistent, dictionary like object. Keys are ordinary strings. Values can be essentially any Python object that pickle can handle, including class instances and recursive structures. By default values are pickled with the default pickle protocol.
That convenience is why shelve shows up in scripts. It looks like a dictionary that survives restarts, so people use it for caches, checkpoints and "where did the last run stop" state.
Where the risk appears
The danger is not in your own process writing and reading its own file. It is in how the file travels:
- A cache directory on a shared drive that several people can write to.
- A shelf copied from a colleague's machine to "save a re-run".
- State files stored in an object bucket with broad write access.
- Fixtures checked into a repository from an outside contributor.
In each case the reader trusts the file's author, and the file format cannot prove who that was.
A note on newer Python
The documentation says that since Python 3.15 you can pass your own serializer and deserializer callables when opening a shelf, and they must be supplied together or an error is raised. That is useful if you want a non pickle format, but check your Python version before relying on it. Older versions do not offer it.
Options for pipeline state
| Need | Safer choice |
|---|---|
| Last processed id or timestamp | A small JSON file or a row in a database table |
| Cached query results | Parquet or CSV with a declared schema |
| Key value cache with several writers | A database or a cache service with access control |
| Arbitrary Python objects | Keep them inside one process, or rebuild them from data |
The principle is to store data, not objects. A file that contains values cannot carry a payload that runs when read.
If you must keep shelves
- Write them to a directory only the job's service account can modify.
- Never read a shelf that arrived from outside the team.
- Add a modification time check, and alert if a state file changes outside a run.
- Delete old shelves when a pipeline is retired, rather than leaving them as clutter that someone may reuse.
How to find them
Search your repositories and storage for imports of shelve and for the extensions your platform's dbm backend creates. Then check who has write access to those locations. The access review is often more telling than the code review.
Where Safeguard fits
Safeguard's code scanning and dependency scanning can help locate risky deserialization calls across repositories, and a free scan is a way to start.
Quick check
Is shelve fine for single user local caching?
If only you can write the file, the risk is low. The problem begins when other people or systems can modify it.
Does a restricted unpickler apply to shelve?
That depends on how you open it. See does a restricted unpickler make pickle safe, and prefer data formats where you can.
Sources
Last reviewed: 19 October 2026.
