GHSA-HPPC-G8H3-XHP3: Out-of-Bounds Read via Unchecked Callback Length in rust-openssl

The rust-openssl ecosystem provides cryptography and TLS functionality by binding to the underlying OpenSSL C library. A critical flaw exists in how the Rust bindings handle Foreign Function Interface (FFI) trampolines for specific cryptographic callbacks. These trampolines serve as the execution bridge between OpenSSL's C-based callback architecture and Rust's safe closure environment.

The vulnerability specifically affects applications utilizing custom Pre-Shared Key (PSK) or TLS/DTLS cookie generation callbacks. When an application implements these custom closures, rust-openssl invokes them during the initial stages of the TLS handshake. The vulnerability manifests when the user-provided Rust closure returns a byte length exceeding the original buffer capacity provided by OpenSSL.

This issue is tracked as CWE-125 (Out-of-bounds Read) and CWE-201 (Insertion of Sensitive Information Into Sent Data). Because the library fails to enforce boundary constraints on the callback's return value, OpenSSL subsequently reads adjacent memory from the process space. An attacker can exploit this condition to extract sensitive data from the heap or stack of the vulnerable service.

The root cause lies in the missing validation within the FFI trampolines located in openssl/src/ssl/callbacks.rs. During a PSK or cookie exchange, OpenSSL invokes a registered C callback, passing a raw byte pointer and a maximum length integer parameter. The rust-openssl trampoline intercepts this call, converts the raw C pointer and length into a mutable Rust slice (&mut [u8]), and delegates execution to the application-defined closure.

The user-provided Rust closure is responsible for populating the slice and returning a Result<usize, ErrorStack> indicating the number of bytes written. Prior to version 0.10.78, the FFI trampoline accepted this usize value and directly cast it to a C-compatible integer type. For PSK callbacks, it cast the value to a u32, and for DTLS cookies, it cast the value to a size_t.

The critical error occurred because the trampoline returned this cast value directly to OpenSSL without verifying that the usize was less than or equal to the capacity of the original slice. If application logic incorrectly reported a length greater than the buffer's true capacity, OpenSSL trusted this return value. OpenSSL then processed the buffer using the inflated length, leading directly to an out-of-bounds memory read during protocol serialization.

The fix implemented in commit 1d109020d98fff2fb2e45c39a373af3dff99b24c addresses the missing bounds check directly within the FFI trampolines. The patch captures the capacity of the provided slice before invoking the user's closure. It then uses pattern matching guards to validate the returned length against this capacity constraint.

// Vulnerable implementation
match (*callback)(ssl, hint, identity_sl, psk_sl) {
    Ok(psk_len) => psk_len as u32,
    Err(e) => {
        e.put();
        0
    }
}

// Patched implementation
let psk_cap = psk_sl.len();
match (*callback)(ssl, hint, identity_sl, psk_sl) {
    Ok(psk_len) if psk_len <= psk_cap => psk_len as u32,
    Ok(_) => 0, // Reject oversize lengths
    Err(e) => {
        e.put();
        0
    }
}

By enforcing the psk_len <= psk_cap condition, the patched code ensures that the trampoline only returns a valid length to OpenSSL. If the closure returns an oversized length, the trampoline falls through to the Ok(_) => 0 branch. Returning zero signals an error to OpenSSL, effectively aborting the handshake and preventing the out-of-bounds read condition entirely.

Exploitation requires the target application to actively use set_psk_server_callback, set_psk_client_callback, or custom cookie generation callbacks. The attacker initiates a standard TLS or DTLS handshake specifically designed to trigger the vulnerable callback path. For example, the attacker might send a ClientHello indicating support for PSK cipher suites or initiating a DTLS connection requiring a HelloVerifyRequest.

The actual trigger depends on the internal logic of the target application's callback implementation. The attacker must supply input parameters (such as the PSK identity hint) that cause the application's closure to return an inflated length. If the application logic contains a bug calculating the length of the copied data, the attacker can force the closure to return a value larger than the allocated slice capacity.

Once OpenSSL receives the inflated length, it constructs the protocol response packet by reading from the original buffer pointer up to the returned length. This operation appends adjacent heap or stack memory to the legitimate buffer contents. The attacker receives this serialized packet and extracts the leaked bytes from the response payload.

The direct impact of this vulnerability is an unauthenticated, remote memory disclosure. Because the flaw resides in the TLS/DTLS handshake phase, the attack occurs before any application-level authentication is established. This pre-authentication status significantly increases the severity of the flaw, exposing the service to uncredentialed external network actors.

The consequences of the memory disclosure mirror the mechanics of the Heartbleed vulnerability, albeit constrained to specific callback configurations. The leaked adjacent memory may contain highly sensitive cryptographic material, including session keys, private certificates, or plain-text credentials residing in the process address space. Repeated exploitation can allow an attacker to systematically map and extract significant portions of the application's memory.

The vulnerability carries a CVSS 3.1 base score of 7.5, reflecting its high confidentiality impact and lack of required privileges. While the availability and integrity of the system remain unaffected, the exposure of cryptographic secrets often leads to complete session compromise. The requirement for a specific application-level bug within the closure implementation prevents universal exploitability across all rust-openssl deployments.

The primary remediation strategy is to upgrade the openssl crate to version 0.10.78 or a subsequent release. This update applies the bounds-checking logic within the FFI trampolines across all affected callback types. Developers utilizing package managers should update their Cargo.toml dependencies and execute cargo update -p openssl to pull the patched version into their build environment.

If an immediate upgrade is technically infeasible, organizations must conduct a rigorous static analysis of their source code. Security teams should audit all instances of set_psk_server_callback, set_psk_client_callback, and cookie generation functions. The audit must verify that every code path within these closures properly validates data bounds and explicitly caps the returned usize value to the length of the provided &mut [u8] slice.

From a detection standpoint, organizations can implement network monitoring rules to identify anomalous protocol messages. Security engineers should configure intrusion detection systems to flag unusually large TLS ServerHello or DTLS HelloVerifyRequest packets originating from endpoints known to use PSK or custom cookies. Fuzzing the callback functions during the integration testing phase provides an additional layer of assurance against localized logic errors.