CVE-2026-40046: Integer Overflow and Protocol Smuggling in Apache ActiveMQ MQTT Decoder

Apache ActiveMQ is an open-source message broker that supports multiple transport protocols, including Message Queuing Telemetry Transport (MQTT). The MQTT transport module allows IoT devices and other clients to connect to the broker using the lightweight publish-subscribe protocol. This module exposes the broker to network traffic, typically over TCP port 1883 or via WebSockets.

CVE-2026-40046 represents a vulnerability in how this MQTT module processes incoming client connections. The flaw is classified as CWE-190 (Integer Overflow or Wraparound). It exists specifically within the packet decoding logic that processes the fixed header of incoming MQTT control packets.

The vulnerability is a regression of a previously disclosed issue, CVE-2025-66168. The Apache Software Foundation patched the 5.19.x branch to address the original flaw, but the commit was not ported forward to the 6.x release line. Consequently, versions 6.0.0 through 6.2.3 remained vulnerable to the exact same attack vector.

The root cause of CVE-2026-40046 lies in the improper implementation of the MQTT specification regarding variable-length integer decoding. Both MQTT v3.1.1 and v5.0 utilize a specific encoding scheme for the "Remaining Length" field within the fixed header of control packets. This field defines the total number of bytes remaining in the packet, encompassing both the variable header and the payload.

The encoding mechanism uses a variable number of bytes, up to a maximum of four. Each byte dedicates its lower seven bits to encode the length value, while the eighth bit functions as a continuation flag. If the Most Significant Bit (MSB) is set to 1, the decoder must read the next byte to continue calculating the length, terminating only when it encounters a byte with the MSB set to 0.

The ActiveMQ MQTT decoder failed to enforce the strict four-byte maximum limit dictated by the protocol specification. The parsing logic read bytes continuously as long as the continuation bit remained set. An attacker supplying a crafted MQTT packet with a Remaining Length field spanning five or more bytes forces the decoder to process it without generating a protocol error.

Processing an oversized length field directly induces an integer overflow condition. The decoder calculates the final length by multiplying the seven-bit values by increasing powers of 128. When the byte count exceeds four, the multiplier or the accumulated value overflows the bounds of a standard 32-bit signed integer in Java, resulting in an incorrect packet length calculation.

The vulnerable logic resided within the ActiveMQ classes responsible for MQTT wire format parsing. The decoder utilized an unbounded loop to extract the Remaining Length value. The loop iteratively read bytes from the network buffer, applying bitwise operations to extract the value and check the continuation bit.

// Vulnerable Implementation (Conceptual)
int multiplier = 1;
int value = 0;
byte digit;
do {
    digit = buffer.readByte();
    value += (digit & 127) * multiplier;
    multiplier *= 128;
} while ((digit & 128) != 0);

This implementation lacks a boundary check on the number of iterations. An attacker sending a sequence of bytes like 0xFF 0xFF 0xFF 0xFF 0x7F forces the loop to execute five times. During the fifth iteration, the multiplier value exceeds the maximum value representable by a 32-bit integer, causing a silent integer overflow.

The patched implementation introduces a counter to track the number of bytes read and enforces the protocol-mandated limit. If the loop executes more than four times, the broker explicitly throws an exception, terminating the malicious connection before any overflow occurs.

// Patched Implementation (Conceptual)
int multiplier = 1;
int value = 0;
int bytesRead = 0;
byte digit;
do {
    digit = buffer.readByte();
    value += (digit & 127) * multiplier;
    multiplier *= 128;
    bytesRead++;
    if (bytesRead > 4) {
        throw new IOException("MQTT Remaining Length field exceeds 4 bytes");
    }
} while ((digit & 128) != 0);

The integer overflow directly facilitates protocol desynchronization between the attacker's client and the ActiveMQ broker. Because the overflow results in an incorrectly small length calculation, the broker assumes the current MQTT control packet ends earlier than it actually does in the TCP stream. The broker stops reading the current packet prematurely.

This desynchronization enables MQTT command smuggling. Any payload data that follows the miscalculated packet boundary remains in the read buffer. When the broker attempts to read the next packet from the connection, it parses this leftover attacker payload. An attacker constructs a payload where the leftover bytes form a perfectly valid, standalone MQTT control packet, such as a PUBLISH or SUBSCRIBE command.

Command smuggling allows the attacker to bypass access controls or state checks that the broker applies during the initial connection sequence. The broker processes the smuggled command in a context that is not fully validated. This technique operates entirely within the application layer and requires no prior authentication if the broker permits anonymous MQTT connections.

The immediate security impact of CVE-2026-40046 includes severe denial-of-service (DoS) conditions. When the integer overflow results in an extremely large or negative length calculation, the broker attempts to allocate memory or access arrays based on this corrupted value. This predictably triggers an ArrayIndexOutOfBoundsException or an OutOfMemoryError in the Java runtime environment.

Repeated exploitation of this flaw crashes the ActiveMQ process entirely or degrades performance to the point of unresponsiveness. Since the attack vector requires minimal bandwidth and only a single crafted packet, it represents an asymmetric DoS vulnerability. The endpoint application exhaustion maps directly to MITRE ATT&CK technique T1499.004.

The capability to inject unauthorized commands into the parsing stream introduces risks beyond simple availability loss. Depending on the specific broker configuration, command smuggling enables unauthorized topic subscription or message publication. Security vendors assess this capability as a high-severity risk, yielding a CVSS v3.1 score of 8.8 in environments where data integrity and confidentiality are strictly regulated.

The primary remediation for CVE-2026-40046 is upgrading the affected software. Organizations running the ActiveMQ 6.x release line must upgrade to version 6.2.4, which includes the necessary loop bounds checking in the MQTT decoder. Users operating on the 5.x branch should verify they are running version 5.19.2 or later to ensure protection against the identical CVE-2025-66168 flaw.

If immediate patching is not feasible, administrators must implement a configuration-based workaround by disabling the MQTT transport connector. This is achieved by editing the activemq.xml configuration file and commenting out or removing the <transportConnector name="mqtt" ... /> directive. This completely mitigates the vulnerability by closing the attack surface, though it breaks functionality for legitimate MQTT clients.

Network-level defenses provide secondary mitigation. Intrusion Prevention Systems (IPS) or MQTT-aware Web Application Firewalls (WAF) can be configured to inspect MQTT packet headers. Rules enforcing strict compliance with the MQTT v3.1.1 and v5.0 specifications—specifically dropping packets where the Remaining Length field exceeds four bytes—will block exploitation attempts before they reach the broker.