CVE-2026-17507 is a signed integer comparison bug in the MLS code of Bouncy Castle for Java, in versions from 1.73 up to but not including 1.86. One small message from a current group member can exhaust the JVM heap and deny service to the rest of the group. It scores 7.5, High. Upgrade to 1.86.
What's affected
Bouncy Castle for Java (bc-java), the org.bouncycastle.mls implementation of the Messaging Layer Security protocol from RFC 9420. The project's own wiki entry for this CVE says:
- Issue affecting: BC before 1.86, from 1.73
- Fixed version: BC 1.86
- Platform: Java 8 and later
The NVD record says "before 1.86" and does not state the lower bound. The project page adds that it starts at 1.73, so that is the range to use. If you do not use the MLS classes, you are not exposed to this particular bug, but you may still want to take 1.86 on its own merits.
The record scores it 7.5 (High) with vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H. One difference in wording is worth knowing. The CVSS vector says no privileges are required, but the project's explanation says the triggering message comes from a current group member. The NVD description says the same. So the practical attacker is a group member, not an anonymous outsider, even though the vector reads otherwise.
How to check if you're exposed
Start by finding out whether you use MLS at all. Most applications that depend on Bouncy Castle use it for ordinary cryptography and never touch the MLS package.
grep -r "org.bouncycastle.mls" src/
mvn dependency:tree -Dincludes=org.bouncycastle
gradle dependencies | grep -i bouncycastle
The first command searches your own source for MLS imports. The second and third list the Bouncy Castle versions in your Maven or Gradle build, including versions pulled in by other libraries. Any version from 1.73 up to 1.85 that is used with MLS is in scope. Also look in shaded or fat jars, because a library can bundle its own copy that your build tool will not list. unzip -l your-app.jar and a search for org/bouncycastle/mls will show it.
What to do now
- Search your code and dependencies for use of
org.bouncycastle.mls, including transitive and shaded copies. - If you use MLS on a version from 1.73 to 1.85, upgrade Bouncy Castle for Java to 1.86 or later and rebuild.
- Run your MLS interoperability or round-trip tests after the upgrade. The project says well-formed leaf indices are unaffected by the fix, so a clean test run is what you expect.
- If you cannot upgrade yet, restrict group membership to parties you trust, because the project says any current group member can send the triggering message.
- Set sensible JVM memory limits and monitoring for services that process group messages, so that a runaway allocation shows up as an alert and not an outage.
Why it matters
The record classes the weakness as CWE-195, Signed to Unsigned Conversion Error. RFC 9420 defines a leaf index as an unsigned 32 bit number. Java has no unsigned int, so Bouncy Castle stored it in a signed int. A value with the top bit set therefore decodes as a negative number. The project is clear that this is a legitimate encoding and has to decode correctly, because the MLS interoperability test vectors use the full range. The bug is in where the range was checked.
Two membership checks compared the decoded value directly against the group's small, positive leaf count. In a signed comparison, every negative number is less than a positive bound, so an out-of-range sender passed. The record classes this as CWE-195, signed to unsigned conversion error. In one path that bad index fed arithmetic that never reaches the root of the tree, appending to a list until the JVM ran out of heap.
The fix widens the value with Integer.toUnsignedLong before the comparison, so an out-of-range index is refused however it was encoded. It is a good reminder for anyone implementing a wire protocol on a platform without unsigned types: range checks belong where the value is used. The record does not report exploitation and CISA does not list the CVE.
Try it yourself
Finding a Bouncy Castle version is harder than it sounds when libraries bundle their own copies. This demo shows how a dependency inventory surfaces direct and transitive versions in one view, which is the first step for a library bug like this one.
Click through it at your own pace, or open it full screen.
Quick check
Am I affected if I only use Bouncy Castle for ordinary encryption?
The bug is in the MLS code. If you never call org.bouncycastle.mls, this particular issue does not apply to you.
Is the attacker an outsider?
The project says the triggering message comes from a current group member, so the attacker must already belong to the group.
Sources
- Bouncy Castle project page for CVE-2026-17507, fetched 18 October 2026
- Fix commit 1eb83471eb01, fetched 18 October 2026
Last reviewed: 18 October 2026.
