Overview of CVE-2026-123543: Unauthenticated Denial of Service via Unbounded Java Deserialization in OpenAM

Introduction

An unauthenticated attacker who can reach an OpenAM server running 16.1.3 or earlier can stall authentication, session validation and policy decisions with only a few small requests carrying serialized Java data. OpenAM usually acts as the central identity service for many applications, so an exhausted node blocks logins and policy agent checks for OpenAM and for everything that depends on it.

OpenAM is an open-source access management server maintained by the Open Identity Platform community. It provides authentication, single sign-on, authorization and federation with OAuth2, OpenID Connect and SAML support. The project describes itself as an independent community continuation of the open-source identity stack from which ForgeRock's commercial products developed, distributed under the CDDL with no per-user or per-server license fees. Public sources do not establish its installed base or market share, but its position as a central authentication and policy decision point makes availability flaws in it relevant to every application it protects.

The NVD record carries a CVSS 3.1 score of 7.5 (High) with vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H, and also records VulnCheck's CVSS 4.0 score of 8.7 (High) with base vector CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N. The CNA classifies the issue as CWE-770 (Allocation of Resources Without Limits or Throttling). The maintainer additionally lists CWE-400 and CWE-502, and is explicit that this issue does not disclose or modify data and does not allow code execution on a default installation. It is an availability problem, and we treat it as one here.

Technical information

Root cause: a class whitelist with no resource limits

Several OpenAM interfaces accept serialized Java data in requests and decode it through a single shared helper, IOUtils.deserialise() in the openam-shared module. In 16.1.3 that helper had a class whitelist and nothing else. It wrapped the bytes, inflated them if they were compressed, and called readObject():

final ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
final ObjectInputStream ois = compressed
    ? new WhitelistObjectInputStream(new InflaterInputStream(bais), classLoader)
    : new WhitelistObjectInputStream(bais, classLoader);

There was no limit on size, depth, reference count or array length. This is a well-known Java deserialization problem. A whitelist stops dangerous gadget classes, but it does not stop harmless, permitted classes from being arranged into expensive shapes. Examples include:

• a small compressed blob that inflates to a huge size
• giant arrays
• nested or back-referenced HashMap/HashSet graphs whose hashCode()/equals() work explodes while they are rebuilt

The result is a large asymmetry between attacker effort and server cost. The maintainer reports that a few small requests can exhaust an OpenAM server. The attack does not depend on high traffic volume.

Why one helper matters

Remote SDK and PLL endpoints, the classic login UI's page session and the admin console all decode serialized Java through IOUtils.deserialise(). That shared code path is why the exposure spans several unrelated-looking surfaces, and also why fixing that one helper fixes every affected endpoint.

Reachable attack surface

The maintainer advisory names three groups of interfaces:

• Legacy remote SDK services: /sessionservice, /authservice and the other PLL (Platform Low Level) and JAX-RPC endpoints used by policy agents and SDK clients.
• Classic login pages under /UI/.
• The administration console.

Protecting only the console therefore leaves the SDK and login surfaces exposed.

Attack flow

The maintainer has deliberately withheld exploitation details to give operators time to upgrade, and no payload, request template or public exploit has been published. Based on the disclosed root cause, the flow looks like this:

1. The attacker locates a reachable affected interface: a remote SDK or PLL endpoint such as /sessionservice or /authservice, a classic login page under /UI/, or the administration console. No authentication is required.
2. The attacker supplies a small serialized Java stream built only from whitelisted classes but arranged into an expensive shape, such as one of those listed above.
3. OpenAM passes the bytes to IOUtils.deserialise(), which inflates them if compressed and calls readObject() with no size, depth, reference or array checks. The class whitelist accepts the stream because every class in it is permitted.
4. Reconstruction consumes disproportionate CPU or heap. Repeating this with a few small requests exhausts the server.
5. OpenAM stops serving authentication, session validation and policy decisions. Downstream, this appears as failed logins, policy agents timing out and session validation errors across the applications that rely on OpenAM.

Patch information

The fix ships in OpenAM 16.1.4, released on 22 October 2026 alongside advisory GHSA-x7wr-5gxg-v833. We could not identify a single fix commit. The release notes link only the full 16.1.3...16.1.4 comparison, so you can verify the relevant changes by reading openam-shared/src/main/java/org/forgerock/openam/utils/IOUtils.java at the two tags.

The patch rewrites deserialise() into several layers of checks, all of which run before an object is rebuilt.

1. Bounded input, including bounded decompression. Compressed data is now inflated up front through a new SizeLimitedInputStream, capped at maxBytes, so a small payload cannot expand without limit. Uncompressed input is checked against the same cap. The size check counts every byte read, so unlike JEP 290's maxbytes it also catches a single huge string or data block.

try (InputStream in = new SizeLimitedInputStream(new InflaterInputStream(new ByteArrayInputStream(bytes)),
        limits.maxBytes)) {
    copyStream(in, inflated);
}

2. A prescan before any class is loaded. SerialStreamScanner.scan(...) reads the raw bytes once, in a single linear pass, before any class is loaded or instantiated. It enforces maxArray, maxExpandedRefs and maxWeakHashKeys. The code comments explain why this layer exists: a JEP 290 filter receives a back reference with serialClass() == null, so it cannot see how much a graph of shared references will cost to rebuild. A rejected stream is logged as a warning and thrown as an IOException.

3. A JEP 290 filter as a second layer. WhitelistObjectInputStream now installs setObjectInputFilter(this::checkInput). The filter rejects anything over maxDepth, maxRefs or maxArray. It also counts how deeply hash and array collections are nested, using the backing array check (Map.Entry[] / Object[]) that HashMap, HashSet and ArrayList make once per instance. Any filter already configured in the JVM (jdk.serialFilter) is still consulted after these checks. Dynamic proxy classes are now refused outright in resolveProxyClass.

4. Sorted collections removed from the whitelist. java.util.TreeSet, java.util.TreeMap and java.lang.String$CaseInsensitiveComparator were on the 16.1.3 fallback whitelist and are removed in 16.1.4. The new source comment explains that a sorted collection's readObject makes no per-instance backing array check, which would make such collections invisible to both the nesting limit and the array length limit.

Default limits and tuning

The defaults live in DeserialisationLimits:

static final long DEFAULT_MAX_DEPTH = 20L;
static final long DEFAULT_MAX_REFS = 100000L;
static final long DEFAULT_MAX_ARRAY = 100000L;
static final long DEFAULT_MAX_BYTES = 10L * 1024 * 1024;
static final long DEFAULT_MAX_COLLECTION_DEPTH = 2L;
static final long DEFAULT_MAX_EXPANDED_REFS = 100000L;
static final long DEFAULT_MAX_WEAK_HASH_KEYS = 32L;

Operators can change these through the openam.deserialisation.limits property, which uses JEP 290-style syntax such as maxdepth=20;maxrefs=100000;maxexpandedrefs=100000. The supported keys are maxdepth, maxrefs, maxarray, maxbytes, maxcollectiondepth, maxexpandedrefs and maxweakhashkeys. The property is read again on every call, so changes take effect without a restart. Invalid or non-positive values are ignored with a warning.

Console versus unauthenticated paths

The admin console's page state legitimately nests collections three deep (map, then map, then set). The patch adds CONSOLE_STATE_MAX_COLLECTION_DEPTH = 3L and an overload that console call sites use to raise the floor for that single limit. Every other limit and the whitelist still apply, and an explicitly configured depth wins either way. The shared JATO decoder intentionally stays on the tighter defaults. In the 16.1.4 Encoder.deserialize, a source comment notes that this path also serves the unauthenticated authentication UI through JATO's own page session handling, so it keeps the default limits while the admin console raises its depth in ConsoleViewBeanBase's deserializePageAttributes override and its own call sites:

return IOUtils.deserialise(b, compressed);

Upgrade notes

• A customization that genuinely needs sorted collections can add them back through openam.deserialisation.classes.whitelist.
• An explicit collection nesting limit below 3 breaks the admin console's page state.
• The official release notes warn that several security fixes in 16.1.4 change behavior that existing deployments may rely on.

Interim controls from the maintainer

Until 16.1.4 is deployed, the advisory recommends:

• At the reverse proxy or firewall, allow /sessionservice, /authservice and other PLL and JAX-RPC endpoints only from hosts running the required policy agents, SDK clients and administration tools.
• Permit administration console access only from administrators' networks.
• Apply request size and request rate limits at the reverse proxy for the affected endpoints and classic login pages under /UI/.
• Run more than one OpenAM server behind a load balancer with health checks, and monitor CPU and heap so an overloaded server can be removed from rotation and restarted.
• Do not expose administrative and agent-facing endpoints to the public internet.

These are interim measures. Small requests can trigger disproportionate work, so request size limits alone cannot close the gap. The software upgrade is the fix.

Detection methods

No CVE-specific signature has been published for CVE-2026-123543: no Nuclei template, Snort/Suricata or Sigma rule, YARA rule, or IoC list. OffSeq's tracker listed the CVE as having no public Nuclei template, no Metasploit or Exploit-DB code and no in-the-wild reports as of 7 November 2026. Active exploitation and threat actor attribution have not been established. With details deliberately withheld, detection relies on three practical approaches.

1. Inventory and dependency checks

The advisory lists these Maven coordinates as affected at <= 16.1.3, fixed in 16.1.4:

• org.openidentityplatform.openam:openam-shared
• org.openidentityplatform.openam:openam-core
• org.openidentityplatform.openam:openam-clientsdk
• org.openidentityplatform.openam:openam-server-only

Include the client SDK in your search. Applications built on the OpenAM client SDK from affected releases are in scope as well as the server WARs. SCA or SBOM tooling that consumes GitHub advisory data should flag these coordinates in Java builds, container images and deployed WAR/JAR files. Inventory checks are the most reliable signal available.

2. Monitor the request paths an attacker must use

Focus reverse proxy, load balancer or WAF logging on /sessionservice, /authservice and other PLL and JAX-RPC endpoints, on /UI/, and on the administration console. Normally only a small, known set of hosts (policy agents, SDK clients and admin tooling) talks to the remote SDK endpoints, so the most useful signal is a request to those paths from a source outside that list, especially from the internet. The same applies to console access from outside administrator networks. Also look for unusual request sizes, request bursts, slow responses and 5xx errors on these specific endpoints. Because a few small requests can be enough, combine size thresholds with source-based and rate-based alerting.

3. Watch for resource exhaustion symptoms

• JVM telemetry: sudden CPU or heap spikes, long garbage collection pauses, or OutOfMemory conditions on OpenAM nodes that do not match login volume.
• Load balancer health check failures or nodes repeatedly leaving rotation.
• Service-level symptoms: failed logins, policy agents timing out, or session validation errors across many applications at once.

Correlating the layers gives the strongest lead. A CPU or heap spike that begins within seconds of requests to the affected paths from a host not on your agent list deserves investigation.

4. What a hostile payload probably looks like

The OpenAM configuration reference documents the 16.1.4 defaults as maxdepth=20;maxrefs=100000;maxarray=100000;maxbytes=10485760;maxcollectiondepth=2;maxexpandedrefs=100000;maxweakhashkeys=32 and describes what the limits cover: deeply nested object graphs (streams nested more than 200 levels are always rejected), collections nested inside collections, oversized arrays and maps with the byte limit applied after decompression, back references that blow up on expansion, hash maps or sets full of unorderable keys sharing hash codes, and collections or maps used as keys, which are always rejected. Those are the shapes to look for in a captured serialized body. Sorted collections such as java.util.TreeSet/TreeMap are no longer on the default whitelist, so their presence in inbound serialized data is also unusual. This is analyst guidance, not a published signature. The documentation does not specify how rejected streams are logged, so do not rely on a particular log message.

All of these methods detect exposure or symptoms rather than the exploit itself. Expect false positives from legitimate load spikes, and expect a careful attack to resemble ordinary small requests.

Affected systems and versions

All artifacts use the group identifier org.openidentityplatform.openam:

• openam-clientsdk: versions through 16.1.3 affected, fixed in 16.1.4
• openam-core: versions through 16.1.3 affected, fixed in 16.1.4
• openam-server-only: versions through 16.1.3 affected, fixed in 16.1.4
• openam-shared: versions through 16.1.3 affected, fixed in 16.1.4

Vulnerable configurations are deployments where any of the following is reachable by an attacker: the legacy remote SDK services (/sessionservice, /authservice and other PLL and JAX-RPC endpoints), classic login pages under /UI/, or the administration console. Applications built on the OpenAM client SDK from affected releases are also in scope.

A record-keeping note: the maintainer advisory displayed "No known CVE" at the time of review, while NVD and VulnCheck both reference GHSA-x7wr-5gxg-v833 for CVE-2026-123543. They describe the same issue. The advisory was published on 22 October 2026, before the NVD record appeared on 8 November 2026.

Vendor security history

This is not the first serialization issue in OpenAM this year:

• CVE-2026-33439 (advisory published 7 April 2026): a critical pre-authentication issue involving unsafe deserialization of jato.clientSession, affecting versions through 16.0.5 and patched in 16.0.6. That was an unfiltered deserialization path with possible command execution, a different failure mode from the present availability flaw.
• OpenAM 16.1.1: addressed a conditional Java deserialization RCE in WebAuthn (CVE-2026-45051) and unsafe deserialization through push notification (CVE-2026-45794).
• OpenAM 16.1.4: addresses this unbounded deserialization denial of service.

Earlier fixes added class restrictions, and this case shows that a whitelist can coexist with a resource exhaustion flaw when size and structure are unbounded. The project also shows active security maintenance. The 16.1.4 notes describe enabling CodeQL security and quality checks, Docker image scanning with Trivy, and GitHub Actions dependency and permission hardening, and the repository supports private advisory reporting. None of the earlier advisories are evidence of exploitation of CVE-2026-123543, and their impact should not be transferred to it.

References

• NVD: CVE-2026-123543
• CVE Record: CVE-2026-123543
• NVD API record for CVE-2026-123543
• GitHub Advisory GHSA-x7wr-5gxg-v833: unauthenticated denial of service through unbounded deserialisation
• GitHub API: GHSA-x7wr-5gxg-v833
• VulnCheck advisory: OpenAM before 16.1.4 Unauthenticated DoS via Unbounded Java Deserialisation
• OpenAM 16.1.4 release announcement
• GitHub release 16.1.4
• GitHub API: release 16.1.4
• Comparison 16.1.3...16.1.4
• GitHub API: comparison 16.1.3...16.1.4
• IOUtils.java at 16.1.4
• IOUtils.java at 16.1.3
• Encoder.java at 16.1.4
• OpenAM configuration reference
• OffSeq Radar entry for CVE-2026-123543
• CISA Known Exploited Vulnerabilities feed
• OpenAM GitHub repository
• OpenAM security overview
• OpenAM security advisories
• GHSA-2cqq-rpvq-g5qj: Pre Authentication RCE via jato.clientSession Deserialization (CVE-2026-33439)
• OpenAM 16.1.1 release announcement
• Open Identity Platform: ForgeRock alternative and migration path
• Open Identity Platform home