CVE-2026-54638: Pre-Authentication Denial of Service via Unbounded Memory Allocation in gotd/td MTProto Parser

The library github.com/gotd/td serves as an open-source Go implementation of the Telegram MTProto protocol, supporting both client-side and server-side operations. A key component of this architecture is the handling of transport-layer handshakes and unencrypted protocol packets. Plaintext messages are typically processed during initial session negotiation and Diffie-Hellman key exchange steps, before cryptographic sessions are fully established and authenticated.

The parsing routing for unencrypted messages is managed by the proto.UnencryptedMessage.Decode method. Because unencrypted handshake packets must be evaluated to establish session states, this decoding endpoint is exposed to any remote, unauthenticated network peer capable of reaching the service port. This exposure creates a direct network attack surface for incoming, untrusted TCP payloads.

The vulnerability is classified as CWE-770 (Allocation of Resources Without Limits or Throttling) and CWE-789 (Memory Allocation with Excessive Size Value). Due to the lack of pre-allocation size verification, an attacker can manipulate the allocation of severe amounts of memory. Successful exploitation disrupts service operations across all established connections, resulting in a complete Denial of Service.

The root cause of CVE-2026-54638 resides in the logical sequence of operations within proto/unencrypted_message.go. During the decoding of a plaintext message, the parser reads an 8-byte authentication key identifier, an 8-byte message identifier, and a 32-bit signed integer representing the payload data length (dataLen). The data length parameter dictates how many bytes of actual payload content follow the packet headers.

Once dataLen is decoded from the incoming buffer via b.Int32(), the application executes a slice expansion routine using the Go built-in allocator. The code executes make([]byte, dataLen) inside an append wrapper to resize the destination byte slice u.MessageData. In Go, allocating a slice via make requires the runtime to request contiguous virtual memory from the allocator and zero-initialize the memory block, an operation that incurs significant CPU and memory overhead proportional to the requested size.

Crucially, the validation of whether the input buffer actually contains dataLen bytes of unparsed network data is deferred to the subsequent execution of the b.ConsumeN method. If the incoming buffer is shorter than the declared length, ConsumeN correctly identifies the anomaly and returns an unexpected EOF error. However, because this boundary verification occurs after the heap allocation has already executed, the application is forced to perform the resource-intensive memory allocation before validating the packet payload completeness. This architectural sequencing creates an asymmetric resource amplification bug.

To evaluate the mechanics of the vulnerability, we analyze the vulnerable sequence in proto/unencrypted_message.go side-by-side with the patched implementation. The vulnerable code path executes the allocation before any physical size assertions can check the buffer limits.

// VULNERABLE PATH
func (u *UnencryptedMessage) Decode(b *bin.Buffer) error {
    // ... (reads auth_key_id and message_id)
    
    dataLen, err := b.Int32()
    if err != nil {
        return err
    }
    
    // VULNERABILITY: Allocation occurs prior to buffer completeness checks.
    u.MessageData = append(u.MessageData[:0], make([]byte, dataLen)...) 
    
    // Boundary validation is executed too late to prevent memory allocation.
    if err := b.ConsumeN(u.MessageData, int(dataLen)); err != nil {      
        return errors.Wrap(err, "consume payload")
    }
    return nil
}

In the patched implementation, the maintainers introduced two defensive boundary assertions prior to calling make. The first check ensures that dataLen is non-negative, preventing integer underflow attacks or unexpected behavior with negative values. The second check compares dataLen against the number of unread bytes physically remaining inside the incoming buffer, represented by b.Len().

// PATCHED PATH
func (u *UnencryptedMessage) Decode(b *bin.Buffer) error {
    // ... (reads auth_key_id and message_id)
    
    dataLen, err := b.Int32()
    if err != nil {
        return err
    }
    
    // Validate that the length is not negative
    if dataLen < 0 {
        return &bin.InvalidLengthError{
            Length: int(dataLen),
            Where:  "plaintext message data",
        }
    }
    
    // Validate that the buffer contains at least the declared number of bytes
    if int(dataLen) > b.Len() {
        return errors.Wrap(io.ErrUnexpectedEOF, "consume payload")
    }
    
    // Allocation is now safe to execute
    u.MessageData = append(u.MessageData[:0], make([]byte, dataLen)...)
    if err := b.ConsumeN(u.MessageData, int(dataLen)); err != nil {
        return errors.Wrap(err, "consume payload")
    }
    return nil
}

By comparing dataLen against b.Len(), the library ensures that the memory allocation is strictly bounded by the actual amount of data transmitted by the peer. Since an attacker cannot force the allocation of more memory than the physical payload size they have already sent over the socket, the asymmetric amplification potential is neutralized.

Exploitation of CVE-2026-54638 requires only a single, malformed TCP packet and no pre-authentication credentials or session state. To execute the attack, the adversary constructs a 20-byte packet containing the MTProto plaintext header structures. The packet layout consists of a 64-bit zeroed auth key ID, a 64-bit arbitrary message ID, and a 32-bit big-endian length parameter.

The attacker sets the final 32-bit length parameter to a massive value such as 0x70000000 (representing approximately 1.75 GB in decimal) and transmits this payload to the target listening TCP port. Because the parser is exposed on the public internet, no transport-layer security or authorization checks intercept this packet prior to the execution of the decoding pipeline.

When the application processes the parsed value, the Go runtime attempts to allocate and zero-initialize a 1.75 GB byte slice on the heap. If the hosting system has a strict virtual memory limit, the operating system kernel instantly sends a SIGKILL to terminate the process via the Out-Of-Memory (OOM) killer. If the allocation succeeds, the runtime is subjected to heavy Garbage Collection (GC) sweeps and memory thrashing, rendering the processor unable to service legitimate connections, which creates a severe service degradation.

The overall security impact of this vulnerability is high, exclusively affecting service availability. The vulnerability is highly reproducible and requires low attack complexity, making it an attractive target for automated disruption campaigns. Because the bug lies on the pre-authentication handshake path, no valid credentials or session keys are required to execute the exploit successfully.

In containerized environments (such as Kubernetes or Docker), the container runtime often enforces strict limits on memory consumption. When the OOM killer terminates the process inside a container, the pod or container is forced into a restart loop. Persistent transmission of these crafted packets can keep the target in a continuous crash loop, dropping all established connections and blocking new initialization requests.

For non-containerized environments, the sudden heap allocation pressure can lead to system-wide instability. If multiple threads concurrently handle such malicious handshake requests, the aggregate allocation requests can crash the host server, affecting other co-located applications and infrastructure.

The primary remediation step is to update the github.com/gotd/td dependency to version 0.145.1 or later. This update modifies the packet decoding logic, verifying that the payload size specified by the remote peer does not exceed the size of the unread incoming buffer before allocating memory.

If immediate dependency upgrades are not possible, administrators should deploy intermediate mitigation controls at the network layer. Configuring a reverse proxy or application firewall that filters out abnormally large unencrypted packets before they reach the backend Go daemon can intercept exploitation attempts. Additionally, implementing rate-limiting on TCP connection establishment restricts the frequency at which an attacker can execute the exploit.

Workload resource enforcement should also be implemented. Configuring strict limits on virtual memory consumption and establishing automatic process restarting protocols (such as health-check probes in Kubernetes) reduces the recovery window following an OOM event. However, because these mitigations only minimize the recovery time rather than preventing the underlying crash, deploying the library patch remains the only permanent resolution.