GHSA-47QP-HQVX-6R3F: Remote Memory Exhaustion (Denial of Service) in JLine3 Telnet Server

The JLine3 project is a Java library designed for handling console input. Its remote terminal capabilities include an integrated Telnet server implementation provided by the org.jline:jline-remote-telnet artifact. This server enables administrators or applications to expose a command-line interface over a TCP port, mimicking a legacy multi-user environment.

During the initial connection phase, the Telnet server handles protocol negotiation via command sequences. This phase occurs before authentication. This design exposes a significant attack surface because any network-facing client can interact with the negotiation parser. The vulnerability lies within the handling of the environment variable configuration option defined in RFC 1572.

Specifically, the server processes incoming environment variables and allocates storage space on the Java virtual machine heap for each pair. Because this parsing engine does not impose boundaries on the quantity of variables, an unauthenticated client can trigger uncontrolled memory allocation. The result is a total denial of service due to memory exhaustion.

The root cause of the vulnerability lies in the lack of collection bounds and state validation during the Telnet NEW-ENVIRON option negotiation. The NEW-ENVIRON protocol (RFC 1572) is designed to allow a client to advertise local environment variables to the server using subnegotiation payloads. The server parses these variables and associates them with the session configuration via the ConnectionData context.

In the vulnerable implementation, the connection metadata manages environment variables within a standard HashMap<String, String> initialized with a default capacity of 20 entries. While the parser imposes length limits on individual variable names (50 characters) and variable values (1000 characters), it completely lacks limits on the aggregate count of elements. This missing architectural constraint allows the HashMap to grow dynamically without restriction.

An attacker can exploit this design by transmitting a continuous, unbounded stream of subnegotiation variables. Because the server parses these sequentially from the raw network socket block inside a loop, it creates new String objects and map nodes for every incoming element. The connection thread remains active and continues reading from the input stream, continuously growing the HashMap until the heap capacity of the JVM is depleted.

To understand the vulnerability, consider the original implementation of the environment negotiation processing loop in TelnetIO.java prior to the fix. The parsing loop runs iteratively without maintaining a tracking index of processed variables:

// Vulnerable Code Path in TelnetIO.java
boolean cont = true;
if (i == NE_VAR || i == NE_USERVAR) {
    do {
        switch (readNEVariableName(sbuf)) {
            case NE_VAR_OK:
                // Inserts directly into the session environment HashMap
                TelnetIO.this.connectionData.getEnvironment().put(str, sbuf.toString());
                break;
            case NE_VAR_UNDEFINED:
                break; 
        }
    } while (cont);
}

The cont flag remains true during the entire lifecycle of the subnegotiation block, and the parsing process only terminates if an error or explicit completion signal occurs. Since the JVM allocates substantial memory headers for each HashMap entry—including Map.Entry wrappers, key/value string object references, and the underlying backing array expansion—small payloads result in high memory overhead.

The remediation introduced by Guillaume Nodet in commit 934f09e6128cee33c2b13d42b6e859c1ee2d194b addresses this issue by declaring a static constraint NE_VAR_COUNT_MAX = 100 and tracking loop iterations:

// Patched Code Path in TelnetIO.java
protected static final int NE_VAR_COUNT_MAX = 100; // Security boundary
 
// ... inside readNEVariables() ...
int varCount = 0; // Local counter
boolean cont = true;
if (i == NE_VAR || i == NE_USERVAR) {
    do {
        switch (readNEVariableName(sbuf)) {
            case NE_VAR_DEFINED:
                LOG.log(Level.DEBUG, "readNEVariables()::NE_VAR_DEFINED");
                if (++varCount > NE_VAR_COUNT_MAX) {
                    // Limit exceeded, logging warning and aborting
                    LOG.log(Level.WARNING, "readNEVariables()::TOO_MANY_VARS (>" + NE_VAR_COUNT_MAX + ")");
                    skipToSE(); // Flush the socket stream safely
                    return;     // Reject further variables
                }
                String str = sbuf.toString();
                // ... remainder of parsing logic ...

This modification enforces a hard ceiling on the number of environment variables accepted during negotiation. If the counter exceeds 100, the parser logs an entry, uses skipToSE() to advance the network stream pointer past the remaining subnegotiation data to prevent desynchronization, and aborts execution. This stops heap exhaustion attempts before they can impact JVM availability.

Exploiting this vulnerability does not require any credentials or pre-authentication steps. The negotiation stage of the Telnet protocol occurs before the user is prompted for credentials or presented with a login shell. An attacker only needs a route to establish a TCP connection to the active Telnet listener.

The attack begins with standard TCP handshake completion. The server offers capability negotiations, including the NEW-ENVIRON option. Once the client indicates support, the server issues a request for environment variables: IAC SB NEW-ENVIRON SEND IAC SE (represented in hex as \xff\xfa\x27\x01\xff\xf0). The attacker responds with an environment data declaration prefix: IAC SB NEW-ENVIRON IS (\xff\xfa\x27\x00).

Instead of terminating the block with the closing command sequence IAC SE (\xff\xf0), the attacker sends a dense, continuous sequence of simulated environment variables. For example, the payload alternates variable declaration types with incrementing byte sequences representing names and values. The parser reads these tokens and populates the JVM heap.

Because of the overhead of Java String allocations, generating 250,000 map entries is sufficient to consume approximately 500 MB of heap. At 13 bytes per variable on the wire, this requires approximately 3.25 MB of payload data. The attack completes quickly and degrades JVM performance until the hosting process crashes.

The vulnerability has a CVSS v3.1 base score of 7.5, reflecting its high impact on application availability. Because JLine3 is designed as an embedded library, the Telnet service runs within the same JVM process as the parent application. Consequently, triggering an OutOfMemoryError in the Telnet thread leads to complete failure of the host process.

The exploitation of this bug results in a full Denial of Service (DoS). The affected system stops responding to remote requests, and any concurrent sessions or unrelated internal operations within the JVM are terminated. In multi-tenant systems or cloud microservices, this can cause cascade failures across related services.

Because this vulnerability requires no authentication and can be triggered via a low-bandwidth stream, it is a highly reliable target for denial of service attacks. However, it cannot be leveraged for remote code execution (RCE) or data confidentiality bypasses. The impact is limited to system availability.

The primary remediation strategy is upgrading the dependency org.jline:jline-remote-telnet to version 4.2.1 or newer. This update integrates the boundary enforcement fix and limits accepted environment variables to 100 per session.

If patching is delayed, administrators should restrict exposure using firewalls or network access control lists (ACLs). Access to the Telnet port must be restricted to trusted, internal IP addresses. Alternatively, if Telnet management is not a business requirement, the module should be disabled entirely.

Detection can be implemented via network-based intrusion detection systems (IDS). Security engineers can deploy custom Snort or Suricata rules to inspect Telnet streams. A rule should watch for long, unbroken sequences of variable negotiation identifiers (\xff\xfa\x27\x00) that exceed normal length thresholds without a corresponding closing sequence (\xff\xf0).