GHSA-vc8p-8pxg-rfwg: Denial of Service via Integer Overflow and Memory Exhaustion in ConnectBot SSH Client Library

The ConnectBot SSH Client Library (commercially managed as cbssh, and distributed via Maven under org.connectbot.sshlib:sshlib) contains a denial of service vulnerability in its parser for Distinguished Encoding Rules (DER). The affected component, which resides in the cryptographic key decoding subsystem, is responsible for processing private-key files encoded in DER or PEM format. When an application attempts to load or authenticate a private key, this parser processes the file's binary ASN.1 structure to extract cryptographic material.

This implementation is exposed to local attack vectors where an application takes private-key input from users, local file paths, or untrusted external storage. If the library parses a specially crafted private key, it triggers an integer overflow and subsequently an unchecked memory allocation request. Because the parser is used during initial authentication configuration and key loading, the exposure is limited to clients and server-side components processing user-supplied key files.

The vulnerability is classified under CWE-190 (Integer Overflow or Wraparound), which propagates into CWE-770 (Allocation of Resources Without Limits or Throttling) and CWE-400 (Uncontrolled Resource Consumption). The vulnerability results in an uncaught OutOfMemoryError, terminating the Java Virtual Machine (JVM) thread or process. Standard cryptographic operations remain uncompromised, but the availability of any system executing this library is fully degraded when processing malformed inputs.

The underlying flaw is located within the logic used to read length indicators in the DER reader implementation (DerReader.kt). When parsing ASN.1 structures under DER rules, elements such as sequences, integers, or octet strings are preceded by an identifier byte and a length indicator. If the length indicator's most significant bit is set, it indicates a long-form length where the lower 7 bits of the initial byte specify how many subsequent bytes represent the actual length.

The vulnerable parser allowed up to 127 length octets to be read without placing a limit on the total number of bytes or the size of the accumulated value. The library accumulated these bytes into a 32-bit signed Kotlin Int accumulator. Because there was no upper bounds checking on this calculation, shifting and logical OR operations caused a signed integer overflow. Specifically, length indicators representing values larger than 2147483647 wrapped around to negative numbers or small positive values.

Furthermore, once the length was parsed, the reader immediately attempted to allocate memory for the data payload. In Kotlin and Java, allocating arrays (such as ByteArray) utilizes the parsed integer directly. Because the parser did not check whether the remaining bytes in the input stream matched or exceeded the declared length, the application attempted to allocate unbounded blocks of memory based entirely on a fabricated length header in a tiny file. This leads to an immediate JVM OutOfMemoryError.

The vulnerable code in DerReader.kt processes lengths through a loop that does not validate the integer boundaries or compare the declared length to the available stream size. Before the patch was applied, the reader accumulated bytes using unchecked logical operations:

fun readLength(): Int {
    val next = data.get().toInt() and 0xFF
    if (next and 0x80 == 0) {
        return next
    } 
    val count = next and 0x7F
    var length = 0
    for (i in 0 until count) {
        val nextByte = data.get().toInt() and 0xFF
        length = (length shl 8) or nextByte
    }
    return length
}
 
fun readInteger(): ByteArray {
    val length = readLength()
    val bytes = ByteArray(length)
    data.get(bytes)
    return bytes
}

The remediation applied in version v0.3.1 addresses this issue by replacing the unchecked accumulator with a 64-bit Long variable and validating both the octet count and the calculated length against the actual buffer size. The updated code behaves as follows:

fun readLength(): Int {
    val next = data.get().toInt() and 0xFF
    if (next and 0x80 == 0) {
        return next
    }
    val count = next and 0x7F
    if (count > 4) {
        throw IOException(\"DER length octet count exceeds 4 bytes\")
    }
    var length: Long = 0L
    for (i in 0 until count) {
        val nextByte = data.get().toInt() and 0xFF
        length = (length shl 8) or nextByte.toLong()
    }
    if (length > Int.MAX_VALUE || length < 0) {
        throw IOException(\"DER length overflow or invalid negative length\")
    }
    if (length > data.remaining()) {
        throw IOException(\"DER length $length exceeds remaining input stream size\")
    }
    return length.toInt()
}

This patch completely resolves the vulnerability because it prevents integer overflow through the use of a Long accumulator, enforces a strict 4-octet ceiling on the length field, and verifies that the remaining payload size matches the declared length before triggering a heap allocation.

Exploitation of GHSA-vc8p-8pxg-rfwg requires that an application use the cbssh library to parse an attacker-supplied DER or PEM-encoded private key. This occurs when an end-user uploads a private SSH key to authenticate an SSH session managed by the client application. Because the parsing happens locally, the attack vector is classified as Local, and the exploit cannot be executed remotely by a malicious SSH server during a connection handshake.

To craft a proof-of-concept payload, an attacker constructs a file containing a malformed ASN.1 sequence. The header specifies an identifier tag (such as 0x02 for an ASN.1 INTEGER), followed by a multi-byte long-form length indicator. By configuring the long-form length bytes to declare an excessive value (for example, 1 GiB) while only supplying a few trailing dummy bytes, the total file size remains under 10 bytes.

The following diagram illustrates the execution flow and failure point when the vulnerable parser processes the malformed key file:

When the application reads the file and invokes decodePemPrivateKey(), the DER parser processes the length declaration and attempts to instantiate a ByteArray. This causes the JVM to seek a contiguous block of heap memory of that size. If the JVM's available heap space is smaller than the requested size, an OutOfMemoryError is immediately thrown, bypassing standard exception catches and causing a process crash.

The security impact of GHSA-vc8p-8pxg-rfwg is primarily concentrated on application availability, leading to a complete Denial of Service (DoS) for the affected thread or process. Because the error thrown is java.lang.Error (specifically java.lang.OutOfMemoryError) rather than a standard java.lang.Exception, typical try-catch blocks targeting standard exceptions do not intercept this event. This causes the error to propagate upward, terminating the active thread, or crashing the hosting process entirely.

There is no direct impact on data confidentiality or integrity. The vulnerability cannot be used to leak sensitive session tokens, read memory contents, or execute arbitrary binary code. It does not undermine the cryptographic strength of successfully negotiated SSH connections, nor does it allow authentication bypass.

According to the CVSS v4.0 calculator, the metric vector evaluates to CVSS:4.0/AV:L/AC:L/AT:N/PR:N/UI:A/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N, yielding a severity score of 6.8. The vulnerability is not documented in the CISA KEV catalog, and there are no reports of active exploitation in the wild.

The primary remediation path is upgrading the cbssh library dependency to version v0.3.1 or higher. This release integrates the necessary bounds checks on length fields within DerReader and adds verification of physical input availability before memory is allocated. For developers using Maven or Gradle, this requires updating the dependency configuration to target the latest stable version of org.connectbot.sshlib:sshlib.

For environments where upgrading the library is not immediately feasible, specific defense-in-depth mitigations should be applied. Applications should implement strict file size validation on any user-provided key files before passing them to the parser. Since a standard RSA or EC private-key file rarely exceeds 16 kilobytes, enforcing a hard limit of 16 KB on input stream buffers blocks payloads attempting large allocations.

Additionally, standard JVM deployment hardening can reduce the impact of local Denial of Service attacks. Configuring the JVM to restart automatically upon critical failure (using flags such as -XX:+OnOutOfMemoryError to trigger recovery scripts) ensures that the service recovers quickly from a process termination. However, these configuration workarounds are secondary to applying the library patch.