To control personal data in pipelines, find it at the source, tag the columns once, and make downstream jobs inherit the tag. Then decide per tag what is allowed: drop, hash, mask or keep with restricted access. Doing this before the first copy is far cheaper than chasing copies afterwards.
Start with the definition
GDPR Article 4 defines personal data as any information relating to an identified or identifiable natural person. The word identifiable matters. A user ID, device ID, IP address or a combination of ordinary fields can qualify even if no name is present. Other laws define terms differently, so check what applies to you with your privacy counsel.
Where personal data hides in a pipeline
- Obvious columns: names, emails, phone numbers, addresses.
- Free text fields: support notes, comments and chat transcripts that people filled with details.
- Identifiers: customer IDs, cookie IDs, device IDs, IP addresses.
- Event payloads: JSON blobs where new fields appear without notice.
- Logs: job logs that print rows on error.
- Staging and temp tables that outlive the job.
- Extracts sent to partners, and exports saved by analysts.
A practical process
- Inventory sources. List every system the pipeline reads, with an owner.
- Sample and classify. Combine name based rules with pattern checks on sampled values. Expect false positives and negatives, so a person confirms the result.
- Tag at the source. Record the classification as metadata on the column, not in a spreadsheet nobody opens.
- Decide handling per tag. For example: direct identifiers never leave the raw layer, quasi identifiers are generalized, free text is excluded unless needed.
- Enforce in code. Transformations apply the handling, and a test fails the build when a tagged column appears where it should not.
- Watch for drift. Schema changes and new JSON keys need a new classification pass.
Free text and JSON need special care
Structured columns are the easy part. Free text can hold anything, and nested payloads can add a field tomorrow. For these, prefer allow lists of known safe keys over block lists of known bad ones.
Lineage makes it tractable
If you can see which tables derive from a tagged column, you can answer where a person's data lives. See what to record for data lineage for the minimum useful set.
Tests worth writing
- A test that scans output tables for patterns like email addresses in columns that should not hold them.
- A test that fails when a new column appears in a raw table without a classification.
- A check that logs from a failing job contain no row values.
Where Safeguard fits
Safeguard offers compliance and evidence automation that helps record controls and reviews, and policies and gates that enforce rules on code changes. Dependency scanning of the repositories behind your pipelines is available through the free scan at https://gold.safeguard.sh/scan.
Quick check
Is a customer ID personal data if there is no name next to it?
It can be. GDPR defines personal data as information relating to an identified or identifiable person, and an ID that can be linked back to someone may qualify.
Why use allow lists for JSON payloads?
New keys appear without notice, so a list of permitted keys fails safe while a list of banned keys fails open.
Sources
Last reviewed: 19 October 2026.
