Juliet is a public collection of test programs for C and C++ that contain known weaknesses and safe variants, hosted by NIST in its Software Assurance Reference Dataset. People use it to check whether a static analysis tool detects certain weakness types. It says little about how a tool performs on your own codebase.
What it contains
The NIST page for Juliet C/C++ version 1.3 describes a collection of test cases in C and C++ organized under 118 different CWEs. It lists 64,099 test cases, an author of the NSA Center for Assured Software, and a submission date in October 2017. The suite is in the public domain, and the page says it is made available under the CC0 1.0 license for foreign rights.
Test case descriptions show the pattern. Each names a weakness, a bad source or sink, and a good variant that is supposed to be safe, for example a case that reads data from a connect socket in one version and uses a hard coded value in the good one.
Why paired cases matter
The pairing is the useful idea. A tool that reports every file as vulnerable would "find" all the bad cases and also fail every good one. Measuring both sides lets you see detection and false alarms together.
If you want to learn the approach on a small scale, write your own pairs. A separate post in this series covers test cases you can write yourself.
What a Juliet result can and cannot tell you
| Question | Can Juliet help? |
|---|---|
| Does the tool understand basic C and C++ weakness patterns? | Yes, as a rough probe |
| Does it work on my web framework in another language? | No, the suite is C and C++ |
| Will it find flaws in my real code at my scale? | No |
| Is the tool fast on a large monorepo? | No, test cases are small |
| Does it handle modern idioms and libraries? | Limited, the suite dates from 2017 |
The suite is synthetic. Real code has layers, frameworks, generated files and history that small test programs do not reproduce.
How to use it sensibly
- Decide that you care about C or C++ first. If your code is in another language, Juliet is the wrong tool for your evaluation.
- Pick a handful of CWEs relevant to you instead of running all 118.
- Compare results across tools using the same case selection and the same build setup.
- Treat the output as one input next to a pilot on a repository you own.
Be skeptical of a vendor number quoted from a public suite without the case selection and method behind it. High scores on a published suite can be tuned for, which is one reason to also test on code the vendor has never seen.
Why it still matters to buyers
Public suites give everyone a common baseline, so a vendor discussion can start from a reproducible exercise instead of marketing claims. Ask the vendor what suite they use internally and whether you can rerun it.
A worked example: reading one paired case
A paired test case has a bad version and a good version of the same operation. Picture a small program that reads a string from a network socket and then copies it into a fixed size buffer. In the bad version, the copy ignores the buffer size, which is a classic overflow pattern. In the good version, the program checks the length first, or uses a copy function that takes a limit.
A useful tool flags the bad version and leaves the good one alone. Now consider what each failure looks like:
| Tool behavior | What it means |
|---|---|
| Flags bad, ignores good | Handles this pattern well |
| Ignores bad, ignores good | Missed the weakness, a false negative |
| Flags bad, flags good | Detects the weakness but cannot see the fix, a false positive |
| Flags neither or both everywhere | Not useful for this class |
Scoring both sides is the point. A tool that flagged every function would catch every bad case and look excellent on a detection only measure, which is why paired cases exist.
A step by step way to run a small slice
- Download the suite from the NIST page and read the documentation, which explains how the cases are organized.
- Choose three to five weakness types that matter to your C or C++ code, for example buffer copy without a size check, use after free, or format string misuse.
- Copy a manageable set of cases for each into a scratch project. Keep the bad and good variants together.
- Make sure the project builds, since many tools need a build to analyze C and C++.
- Run each candidate tool with default settings first, then with the settings the vendor recommends.
- Record, for every case, whether the bad one was flagged and whether the good one was flagged.
- Summarize per weakness type, not as a single percentage.
Keep the same cases and settings for each tool you compare, otherwise the comparison is not fair.
Limits you should state in any write up
The suite is synthetic and small per case. The NIST page lists 64,099 test cases across 118 weakness types for version 1.3, which sounds large, but each case is a short program written to demonstrate one pattern. Real code has long call chains, macros, third party headers and unusual build systems. Results on the suite tell you whether the engine understands basic patterns, and nothing about how it behaves on your monorepo.
The suite is also public. A vendor can tune an engine to it, either on purpose or as a side effect of using it for development. That is why a result from a public suite should always be followed by a trial on your own code.
Where this fits with your other evidence
| Evidence | What it tells you |
|---|---|
| Public paired test suite | Basic pattern handling, comparable across tools |
| Seeded bugs in your own code | Whether the tool handles your frameworks and build |
| A pilot on a real repository | Noise, speed and developer experience |
| Reference conversations | How other teams fared over months |
Use all four if you can, and weigh the pilot most heavily. A related post on seeding bugs in your own code shows how to build the second row.
Other languages
If your code is not C or C++, look for a test suite built for your language, or build your own paired cases. The method transfers even when the cases do not: pick a weakness, write a bad and a good version, and score both.
Try it yourself
Click through it at your own pace, or open it full screen. Look at what a finding shows, because a paired test checks that the good version produces no such finding.
Quick check
Does a perfect score on Juliet mean the tool is accurate?
No. It means the tool does well on that synthetic set. Your code, language and framework may behave differently.
Is Juliet useful for Java or JavaScript?
The NIST entry described here covers C and C++. For other languages, look for a suite designed for them or build your own cases.
Can I use Juliet to compare two SAST tools fairly?
Yes for C and C++ basics, if you use the same cases, build setup and settings for both. Add a trial on your own code before you decide.
Safeguard offers SBOM and dependency scanning, and a free scan is available at gold.safeguard.sh/scan.
Sources
Last reviewed: 19 October 2026.
