Compute a SHA-256 hash of every dataset file when you first accept it, store the values in your repository, and compare them each time the data is reused or moved. A checksum proves the file has not changed since you recorded it. It does not prove the data was clean to begin with.
What a checksum does and does not tell you
A hash is a short fingerprint of a file's exact bytes. Change one byte and the fingerprint changes. That makes it a good tamper and corruption detector between two points in time.
It cannot tell you whether the content was poisoned, mislabeled or unlicensed before you hashed it. Think of it as locking the door after you have inspected the room.
What a popular loader checks
The Hugging Face Datasets documentation lists the integrity checks its load function performs: the number of splits, the number of samples in each split, the list of downloaded files, and SHA256 checksums of the downloaded files. The page states that the checksum verification is disabled by default. It also says that a verification error likely means the dataset author changed the data files, and that you can bypass checks by setting the verification mode to no checks.
Two lessons follow. Do not assume checksums are being verified just because a loader mentions them. And never pass the bypass option to silence an error without finding out why the data changed.
Step by step
- Collect. Download the files into a staging area, not straight into a training folder.
- Inspect. Sample records, check counts and look at obvious outliers, duplicates and sensitive content.
- Hash. Run a SHA-256 tool on each file. On many systems that is the sha256sum command, or shasum with the 256 option on macOS.
- Record. Save a manifest with file name, size, hash, source address and date, and commit it.
- Store. Keep the approved files in storage that your pipeline can read but that few people can write.
- Verify on use. Before each training or indexing job, recompute and compare hashes. Fail the job on any mismatch.
- Re approve on change. A new version is a new dataset. Repeat steps 2 to 4.
A manifest can be as simple as this.
file,size_bytes,sha256,source,date
train.parquet,104857600,<hash>,example.org/data,2026-10-19
Sharded and large datasets
Large datasets split into many files. Hash each shard, then hash the manifest or store it in a signed commit so the list itself cannot be edited quietly. If you use a versioned data store, record its version identifier too, but keep your own hashes as an independent check.
Common mistakes
- Taking the checksum from the same page that serves the file. If an attacker can change the file they can change the number. Use a value from a second channel, or your own first hash.
- Hashing the compressed archive but loading an extracted copy that is never checked.
- Rehashing after a conversion step and forgetting that the converted file is a new artifact that needs its own record.
- Skipping the check for small files because they seem harmless.
Where this fits
Checksums are one control in a longer chain: provenance, review, storage control and re verification. For the poisoning risks they help with, see what split view and frontrunning poisoning mean.
Quick check
Is MD5 good enough for dataset files?
Use SHA-256. Older algorithms have known weaknesses, and there is no cost to choosing the stronger one.
Does a matching checksum mean the dataset is safe?
No. It means the file matches what you recorded. Review the content before you record it.
Sources
Last reviewed: 19 October 2026.
