CVE-2026-56818: Denial of Service via Memory Pinning in Netty Redis Array Aggregator

Netty's Redis codec offers high-performance parsing and aggregation for the Redis Serialization Protocol (RESP). The RedisArrayAggregator component converts incoming sequential RESP elements (such as bulk strings or integers) into structured, composite array structures represented by ArrayRedisMessage objects.

To manage hierarchy and nested structures, the aggregator tracks incoming components using an internal stack of aggregate states. To prevent excessive memory allocation by untrusted peers, the handler enforces configured security limits, specifically bounding both the maximum element count (maxElements) and the maximum nesting depth (maxNestedArrayDepth).

A critical vulnerability exists where the handler validation logic deviates across different threshold checks. When specific constraints are violated, the aggregator raises decoder exceptions without releasing the already gathered partial states or decrementing the reference count of associated pooled buffers. This behavior leaves allocated resources active on the handler context, introducing a remote resource-pinning vector.

The root cause is located in the RedisArrayAggregator.decodeRedisArrayHeader method during the assessment of incoming array length fields. The handler uses an internal stack named depths containing AggregateState instances that retain reference counts of previously accumulated child payloads.

Under normal execution, if the incoming array structure exceeds the maximum allowed nesting limit (depths.size() >= maxNestedArrayDepth), the code executes releaseAndClearDepths(), which releases all retained direct buffers, and then raises a CodecException.

However, if the incoming header specifies an array length that exceeds the maxElements boundary, or if a negative length (bad length) check is triggered, the system directly instantiates and throws a CodecException without invoking releaseAndClearDepths(). This oversight allows references to accumulated buffers to remain pinned in JVM memory, as the internal stack is never systematically cleared on these exception paths.

The original implementation checked limits in sequence but failed to clear the accumulation stack on every path that raised an error.

// Vulnerable logic in RedisArrayAggregator.java
private RedisMessage decodeRedisArrayHeader(ArrayHeaderRedisMessage header) {
    if (header.isNull()) {
        return ArrayRedisMessage.NULL_INSTANCE;
    } else if (header.length() > 0L) {
        if (header.length() > maxElements) {
            // State is not cleared before throwing this exception
            throw new CodecException("this codec doesn't support longer length than " + maxElements);
        }
        if (depths.size() >= maxNestedArrayDepth) {
            releaseAndClearDepths(); // Only cleared here
            throw new CodecException("max nested array depth exceeded: " + maxNestedArrayDepth);
        }
        depths.push(new AggregateState((int) header.length()));
        return null;
    } else {
        // State is not cleared before throwing this exception
        throw new CodecException("bad length: " + header.length());
    }
}

The remediation introduces a centralized helper method, clearAndCreateException, to ensure that state cleanup is applied consistently across all validation pathways:

// Remediation logic in RedisArrayAggregator.java
private CodecException clearAndCreateException(String msg) {
    releaseAndClearDepths(); // Explicitly releases all pooled memory and clears the depths stack
    return new CodecException(msg);
}
 
private RedisMessage decodeRedisArrayHeader(ArrayHeaderRedisMessage header) {
    if (header.isNull()) {
        return ArrayRedisMessage.NULL_INSTANCE;
    } else if (header.length() > 0L) {
        if (header.length() > maxElements) {
            throw clearAndCreateException("this codec doesn't support longer length than " + maxElements);
        }
        if (depths.size() >= maxNestedArrayDepth) {
            throw clearAndCreateException("max nested array depth exceeded: " + maxNestedArrayDepth);
        }
        depths.push(new AggregateState((int) header.length()));
        return null;
    } else {
        throw clearAndCreateException("bad length: " + header.length());
    }
}

An unauthenticated attacker can exploit this state retention vulnerability through a multi-step sequence over an established TCP connection.

First, the attacker initiates a valid array sequence to trigger memory allocation on the server side, sending a small header and a large bulk string. The aggregator allocates a pooled direct buffer and pushes the state to the stack.

Second, the attacker sends a subsequent array header with a length value intentionally exceeding the configured maxElements limit. This triggers a validation failure on the server.

Third, because the state cleanup is bypassed, the server leaves the allocated direct buffer pinned in memory. The attacker keeps the socket connection open and repeats this sequence, forcing the server to exhaust its pooled direct memory space.

The consequence of this vulnerability is a denial of service (DoS) caused by persistent JVM memory exhaustion. Because high-performance Netty servers utilize a pooled memory model (such as PooledByteBufAllocator), unreleased direct buffers do not return to the system and cannot be reclaimed by standard garbage collection.

The retention of these buffers is tied directly to the lifetime of the channel handler context. If the application continues processing traffic on the same TCP channel or catches decoder exceptions without tearing down the connection, memory remains pinned.

By repeatedly inducing this error state across multiple sessions, an attacker can consume all available direct or heap memory allocations. This forces the system to throw an OutOfMemoryError on subsequent requests, causing the Netty worker threads to terminate or crash the application.

The primary defense is updating the Netty dependency to a secure version. The patch has been backported and released in versions 4.1.136.Final and 4.2.16.Final.

If immediate library upgrades are not feasible, implement a strict connection-teardown routine in the pipeline. Ensure that your Netty pipeline handler explicitly catches CodecException and closes the associated channel to free up handler resources.

@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
    if (cause instanceof CodecException) {
        // Force connection termination to release any pinned aggregate states
        ctx.close();
    } else {
        ctx.fireExceptionCaught(cause);
    }
}

Additionally, monitor application performance telemetry. Look for continuous linear growth in direct memory pools alongside unhandled exception logs referencing Redis decoder limit failures.