CVE-2026-61634: Heap Memory Exhaustion in RabbitMQ Java Client

The vulnerability identified as CVE-2026-61634 lies within the RabbitMQ Java Client (rabbitmq-java-client) prior to version 5.33.0. The library represents the official Java client for interacting with RabbitMQ brokers using the AMQP 0-9-1 protocol. The flaw resides in how incoming AMQP frames are validated relative to negotiated limits.\n\nDuring a normal AMQP 0-9-1 handshake, the client and broker negotiate a maximum permissible frame size called frame_max. In affected versions of the library, the socket frame reader and Netty transport handlers did not dynamically bind their internal allocation limits to this negotiated threshold. Instead, they compared incoming frame sizes against the pre-configured maximum message body limit (maxInboundMessageBodySize).\n\nBecause this message limit is significantly larger than standard negotiation limits, a malicious broker can send a frame that exceeds the negotiated frame_max but remains under the message size cap. The client then attempts to allocate a byte array proportional to the size specified in the frame header. This unchecked memory allocation can cause the client JVM to exhaust its heap, resulting in a Denial of Service condition.

The root cause of CVE-2026-61634 is an improper input validation flaw (CWE-20) during and after the AMQP connection handshake. When establishing a connection, the AMQP 0-9-1 protocol follows a strict sequence of negotiations. The broker sends a Connection.Tune frame specifying the maximum channel count, frame size, and heartbeat interval. The client is then supposed to enforce these parameters for all subsequent frames.\n\nHowever, the unpatched Java client maintained a static limit for inbound frame payload checks. The internal framing handlers, including SocketFrameHandler and Netty's pipeline, utilized the maxInboundMessageBodySize property to validate incoming frames. By default, this property is set to tens of megabytes, whereas the negotiated frame_max is typically limited to a few kilobytes (e.g., 4096 bytes).\n\nAn attacker-controlled broker can exploit this discrepancy during or after the handshake by transmitting a frame with an inflated size field in its header. Upon receiving this header, the client reads the 32-bit payload size, verifies it against the high maxInboundMessageBodySize threshold, and instantly invokes 'new byte[payloadSize]'. If the payload size is set to a large value (such as 16MB or greater), the JVM attempts to allocate a contiguous block of heap memory, triggering a java.lang.OutOfMemoryError before checking whether the frame violates the protocol.\n\nmermaid\ngraph LR\n subgraph Client [\"Vulnerable JVM Client\"]\n Socket[\"Socket Stream\"] --> Header[\"Read Frame Header (payload_size=16MB)\"]\n Header --> Check{\"payload_size < maxInboundMessageBodySize (64MB)?\"}\n Check -- Yes --> Alloc[\"Allocate 'new byte[16777216]'\"]\n Check -- No --> Fail[\"Reject Frame\"]\n Alloc --> OOM[\"java.lang.OutOfMemoryError (Crash)\"]\n end\n Broker[\"Malicious Broker\"] -- \"Sends 16MB Frame Header (negotiated limit 4KB)\" --> Socket\n

The fix for CVE-2026-61634 was distributed across three primary commits targeting the connection management, frame reading, and Netty transport pipelines. First, in AMQConnection.java, the client was modified to dynamically propagate the negotiated frame_max value to the inbound FrameHandler once the connection tuning phase completes.\n\njava\n// Patched AMQConnection.java\n@@ -430,6 +430,14 @@ public void start()\n connTune.getFrameMax());\n this._frameMax = frameMax;\n \n+ // Bound inbound frames to the negotiated frame_max. EMPTY_FRAME_SIZE is\n+ // the per-frame overhead; +1 because the reader rejects payloads >= the limit.\n+ if (frameMax > 0) {\n+ _frameHandler.setMaxInboundFramePayloadSize(\n+ Math.min(this.maxInboundMessageBodySize,\n+ frameMax - AMQCommand.EMPTY_FRAME_SIZE + 1));\n+ }\n+\n int negotiatedHeartbeat =\n negotiatedMaxValue(this.requestedHeartbeat,\n connTune.getHeartbeat());\n\n\nSecond, to address the pre-negotiation phase where the client reads handshake frames before frame_max is established, the client was modified to initialize the socket handler with a minimal safe limit (AMQP.FRAME_MIN_SIZE = 4096 bytes). In Frame.java, the validation check was refactored to use Utils.enforceFrameMax prior to allocating any memory arrays.\n\njava\n// Patched Frame.java\n@@ -106,7 +108,7 @@ public static Frame fromBodyFragment(int channelNumber, ByteBuffer body, int off\n *\n * @return a new Frame if we read a frame successfully, otherwise null\n */\n- public static Frame readFrom(DataInputStream is, int maxPayloadSize) throws IOException {\n+ public static Frame readFrom(DataInputStream is, IntSupplier payloadLimit) throws IOException {\n int type;\n int channel;\n...\n channel = is.readUnsignedShort();\n- int payloadSize = is.readInt();\n- if (payloadSize < 0 || payloadSize >= maxPayloadSize) {\n- throw new MalformedFrameException(format(\n- \"Frame body size is invalid (%d), maximum configured size is %d. \" +\n- \"See ConnectionFactory#setMaxInboundMessageBodySize \" +\n- \"if you need to increase the limit.\",\n- payloadSize, maxPayloadSize\n- ));\n- }\n- byte[] payload = new byte[payloadSize];\n+ int frameSize = is.readInt();\n+ Utils.enforceFrameMax(frameSize, payloadLimit.getAsInt());\n+ byte[] payload = new byte[frameSize];\n is.readFully(payload);\n\n\nFinally, the Netty transport pipeline handler (NettyFrameHandlerFactory.java) was updated to dynamically recreate Netty's LengthFieldBasedFrameDecoder with the negotiated frameMax value. This ensures that both standard blocking I/O and Netty-based connections strictly and dynamically enforce the negotiated frame limits at runtime.

Exploitation of CVE-2026-61634 requires the client application to establish a connection to an untrusted or compromised AMQP broker. The threat model includes scenarios where the application connects to a public-facing broker or where a Man-in-the-Middle (MITM) attacker can hijack the connection and act as a proxy.\n\nOnce the socket connection is initiated, the broker begins the standard AMQP handshake. It sends a Connection.Start frame and eventually a Connection.Tune frame. In this tuning frame, the broker dictates a small frame_max limit (e.g., 4096 bytes). The client accepts this parameter.\n\nDirectly after the tuning phase, the broker sends an oversized protocol frame, such as an artificially padded Connection.Open-Ok frame. The frame header specifies a payload_size of several megabytes. Because the unpatched client has not yet updated its internal handler boundaries to the 4096-byte limit, it parses the header, validates it against maxInboundMessageBodySize, and attempts memory allocation. This causes immediate exhaustion of the JVM heap space, crashing the application.

The security impact of CVE-2026-61634 is classified as a client-side Denial of Service (DoS). By causing a java.lang.OutOfMemoryError within the JVM, an attacker can reliably terminate the execution of the client application. In microservice architectures, this can lead to a cascading failure across dependent systems.\n\nThe vulnerability is rated with a base CVSS score of 0.0 by the initial CNA publishing, but practically acts as a Medium-to-High severity risk. The attack complexity is elevated because the attacker must control or compromise the AMQP broker or intercept the network connection. However, no client-side privileges are required, and the exploitation requires no user interaction.\n\nAdditionally, because the vulnerability triggers a heap exhaustion, it can disrupt other critical processes running inside the same JVM container. In multi-tenant environments or cloud-native applications, an OutOfMemory crash of a shared JVM instance affects all hosted applications, magnifying the overall impact.

The recommended remediation for this vulnerability is to upgrade the com.rabbitmq:amqp-client dependency to version 5.33.0 or later. This version contains the complete set of patches that enforce tight frame boundaries during both pre-negotiation and runtime execution phases.\n\nIf upgrading the dependency is not immediately feasible, organizations should implement workarounds to reduce the attack surface. Specifically, developers can limit the maximum possible heap allocation on invalid frames by manually reducing the default client-side maximum inbound message body size via the ConnectionFactory API.\n\njava\nConnectionFactory factory = new ConnectionFactory();\n// Restrict the maximum frame payload validation ceiling to a safer, smaller size (e.g., 2MB)\nfactory.setMaxInboundMessageBodySize(2 * 1024 * 1024);\n\n\nAdditionally, security teams should implement Transport Layer Security (TLS) with peer verification to ensure the client only connects to authenticated, trusted brokers. This prevents MITM hijackers from presenting fake handshakes and injecting malicious oversized frames.