spark.redaction.regex tells Spark which configuration properties and environment variables in driver and executor environments hold sensitive values, and hides those values in the environment UI and in logs such as YARN and event logs. The default pattern is short, so secrets with unusual names can still show.
What the default matches
According to the Spark configuration reference, the default is a case insensitive match on four words: secret, password, token, and access key (with an optional dot between the two words). When the regex matches a property key or value, the value is redacted from the environment UI and various logs.
Notice what that means. A property called db.password is covered. A property called db.pwd is not. Neither is conn.string, even if the string contains a password, unless the value itself matches the pattern. Names chosen by your team decide whether the protection applies.
Related settings
The same page documents two neighbours. spark.redaction.string.regex decides which parts of strings Spark produces contain sensitive information and replaces them with a dummy value, which is used for the output of SQL explain commands. And a SQL specific setting covers option keys in a command's options map, with a default that matches the word url, applied on top of the global redaction.
So there are three layers: property names and values, parts of generated strings, and SQL command options. Each has its own pattern.
A quick test on your own cluster
You do not need to trust the pattern, you can test it.
- Pick a non-production application.
- Set a throwaway property whose name you would plausibly use, for example a vendor specific key name.
- Start the application and open the environment tab in the UI.
- Check whether the value shows as redacted.
- Check the event log and driver log for the same value.
- Repeat with the names your teams actually use for credentials.
Anything that shows in clear text needs either a rename or a wider pattern.
Widening the pattern, carefully
You can set spark.redaction.regex to something broader that includes your own naming conventions. Two cautions. First, test it, because an invalid or overly greedy pattern changes what operators can see when they debug. Second, keep the default words in your replacement, since setting the property replaces the default rather than adding to it.
A better habit is to stop passing secrets as Spark properties at all. Where your platform offers a secret store, reference the secret by name and have the job read it at the point of use. See Databricks secret scopes and redaction limits for how one platform handles this, and Spark security settings to check for related defaults.
What redaction does not do
Redaction is a display and logging filter. It does not encrypt the value, restrict who can submit jobs, or stop a job from printing a secret it read. A line of application code that logs a connection string will not be rescued by a regex on property names. Treat redaction as a safety net, not the control.
Safeguard's dependency scanning is a separate layer: it tells you whether libraries in your jobs have known vulnerabilities. You can try the free scan at gold.safeguard.sh/scan.
Quick check
Does the default pattern cover a property named db.pwd?
Not by name. The default matches secret, password, token and access key. A shorter name needs a wider regex or a rename, unless the value itself matches.
Does redaction stop my own code from logging a secret?
No. It applies to configuration properties and environment variables shown in the UI and logs, not to values your code prints.
Sources
- Spark Configuration, Apache Spark documentation, fetched 19 October 2026.
Last reviewed: 19 October 2026.
