CWE-172 is a class level weakness: the product does not properly encode or decode data, and the result is unexpected values. Encodings exist so data can travel safely through a medium. When code decodes too many times, too few times, or in the wrong order against its checks, the value the check approved is not the value that gets used.
Web applications, URL routers, file servers and API gateways are the usual homes. Every layer that percent decodes, unescapes entities or converts character sets is a place this can happen.
How it happens
The pattern is almost always an ordering problem. A security check inspects the input in one form. A later step converts it to another form. The attacker supplies text that is harmless in the first form and harmful in the second.
The observed examples show different flavors.
- CVE-2004-1315 is forum software that improperly URL decodes the highlight parameter when extracting text to highlight. Remote attackers could execute arbitrary PHP code by double encoding the value, so that a second decoding produced the dangerous form.
- CVE-2004-1939 is an XSS protection mechanism that tries to remove the slash that could close tags, but is bypassed using double encoded slashes, written %252F.
- CVE-2001-0709 is a server that lets a remote attacker obtain source code of ASP files via a URL encoded with Unicode.
- CVE-2005-2256 lists hex encoded path traversal variants such as %2e%2e, %2e%2e%2f and %5c%2e%2e.
Here is the order bug in a few lines.
from urllib.parse import unquote
# Vulnerable: validates the encoded form, uses a decoded form
def read_page(param: str) -> bytes:
if ".." in param:
raise ValueError("bad path")
path = unquote(param) # %2e%2e becomes ..
return open("/srv/pages/" + path, "rb").read()
The string %2e%2e/%2e%2e/etc/passwd has no two dots in a row when it is checked. After decoding it does.
What an attacker gains
MITRE lists integrity impact as unexpected state. The examples show what that has meant in practice: code execution in the forum case, source disclosure in the Unicode case, and filter bypass and file traversal in the others. Encoding bugs rarely carry a payload of their own. They carry another attack past the guard.
How to find it
MITRE lists no detection methods for this class, so test and review.
- Send each sensitive character in several forms: plain, percent encoded, double percent encoded, and Unicode forms. Include dots, slashes, backslashes, quotes and angle brackets.
- Map every decode step in the request path: web server, framework, router, your code, and downstream services. Write the order down. Any validation that runs before a later decode is a candidate.
- Look for code that decodes a value and then decodes the result again, directly or through a library call applied twice.
- Run dynamic and API scans against a staging copy. They send many encoded variants quickly.
How to fix it
The mitigation guidance asks for a strict input validation strategy.
- Assume all input is malicious. Use an accept known good strategy: a list of acceptable inputs that strictly conform to the specification. Reject anything that does not conform, or transform it into something that does.
- Consider every relevant property when validating: length, type, the full range of acceptable values, missing or extra inputs, syntax, consistency across related fields and business rules.
- Decode input once, into the internal representation the application uses, and validate that. Never validate the encoded form and then decode.
from urllib.parse import unquote
import os
def read_page(param: str) -> bytes:
path = unquote(param) # decode exactly once
base = os.path.realpath("/srv/pages")
full = os.path.realpath(os.path.join(base, path))
if os.path.commonpath([base, full]) != base:
raise ValueError("bad path")
return open(full, "rb").read()
- After the single decode, reject values that still contain encoded forms where none are expected. A leftover percent sequence after decoding is a sign of double encoding.
- Prefer identifiers over names. Map a short key to a file on the server side and do not accept paths at all.
- Make sure all layers agree on the encoding, and configure servers to reject non canonical forms of URLs.
Try it yourself
An API scan sends encoded variants at endpoints and shows which responses differ. Click through a sample run.
Click through it at your own pace, or open it full screen.
Dependencies that decode for you
Several of the observed cases are in forum software, servers and filters, which are the kind of components you pull in instead of write. The free in-browser dependency scan checks up to 100 dependencies from a manifest or lockfile, needs no login and keeps the file on your machine. Reachability analysis shows whether your code actually calls the affected function.
Quick check
Why does double encoding bypass a filter?
The filter decodes once and sees harmless text. A later step decodes again and produces the dangerous character. CVE-2004-1939 used %252F, a double encoded slash.
Is blocking two dots in a row enough to stop traversal?
Not when the value can still be decoded after the check. CVE-2005-2256 lists hex encoded variants of the traversal sequence.
Where should validation sit relative to decoding?
After a single decode, on the form the application will actually use.
