RustFS Leak: When Error Logs Become Credentials

In the world of distributed systems, we often obsess over encryption in transit, encryption at rest, and fancy key management services. We build Fort Knox, install retina scanners, and hire armed guards. Then, just to be helpful, we tape the combination code to the forehead of the guard standing outside.

That is essentially what happened with RustFS, a distributed object storage system built in Rust. In an attempt to make debugging authentication errors easier, the developers inadvertently created a mechanism that dumps the keys to the kingdom straight into the system logs. This isn't a complex heap overflow or a race condition; it's a logic error born of good intentions and bad OpSec.

While the CVSS score sits at a deceptive 'Low' (2.9)—primarily because the attacker needs access to logs—anyone who has worked in a DevOps environment knows that kubectl logs permissions are handed out like candy. If your logs are shipped to ELK, Datadog, or Splunk, that 'Low' severity quickly transforms into a 'Critical' nightmare.

The vulnerability lives in crates/ecstore/src/rpc/http_auth.rs, specifically within the signature verification logic. RustFS uses HMAC (Hash-based Message Authentication Code) to verify that Remote Procedure Calls (RPCs) are legitimate. The server calculates an expected_signature using a shared secret and compares it to the incoming request's signature.

So far, so good. But what happens when they don't match? In a secure system, the server should return a generic 401 or 403 error. Maybe log 'Signature verification failed for user X'.

In RustFS, the code decided to be extremely helpful. It thought, "Hey, if the signatures don't match, the developer probably wants to know why." So, it logged everything involved in the calculation. Everything. Including the secret itself. It's the cryptographic equivalent of a bank teller shouting your PIN out loud because you typed it wrong the first time.

Let's look at the diff. It's rare to see a vulnerability so obvious that it makes you wince physically. Here is the vulnerable logic in verify_rpc_signature:

// The Vulnerable Logic
if signature != expected_signature {
    error!(
        "verify_rpc_signature: Invalid signature: secret {}, url {}, method {}, timestamp {}, signature {}, expected_signature {}",
        secret, // <--- OOPS.
        url, 
        method, 
        timestamp, 
        signature, 
        expected_signature
    );
    return Err(std::io::Error::other("Invalid signature"));
}

The fix, implemented in commit 6b2eebee1d07399ef02c0863bd515b4412a5a560, is straightforward: stop logging the secret. The developers also realized that logging the full expected_signature allows for offline brute-forcing or verification of the secret, so they masked that too.

// The Fix
if signature != expected_signature {
    error!(
        "verify_rpc_signature: Invalid signature: url {}, method {}, timestamp {}, signature {}, expected_signature: {}***{}|{}",
        url,
        method,
        timestamp,
        signature,
        // Masking the expected signature to show only start/end chars
        expected_signature.chars().next().unwrap_or('*'),
        expected_signature.chars().last().unwrap_or('*'),
        expected_signature.len()
    );
    return Err(std::io::Error::other("Invalid signature"));
}

As a bonus, they also caught a similar issue in rustfs/src/config/mod.rs. The configuration struct Opt was using a default #[derive(Debug)]. When the app started up or crashed, it would pretty-print the entire configuration state, including secret_key and kms_vault_token. They replaced this with a manual fmt::Debug implementation that sanitizes these fields.

Exploiting this requires zero coding skill and only basic access privileges (specifically, the ability to read logs). The attack flow is embarrassingly simple:

1. Poke the Bear: Send a request to the RustFS RPC endpoint with a completely made-up signature. The server tries to verify it, fails, and triggers the error! macro.
2. Read the Receipt: Open your log aggregator (or docker logs). Look for the string "verify_rpc_signature".
3. Profit: Copy the plain-text secret from the log entry.

Once the attacker has the HMAC secret, they are effectively a super-admin. They can sign requests to read files, delete volumes, or reconfigure the cluster. The system cannot distinguish them from a legitimate node.

The CVSS 4.0 score of 2.9 (Low) is technically correct because the vulnerability requires "Attack Requirements: Present" (access to logs). However, in the real world, this score is dangerous. It assumes that logs are treated as highly sensitive data, which is rarely the case.

In modern microservices architectures, logs are often piped to third-party services, viewed by junior developers, or stored in S3 buckets with permissive read policies. If an internal bad actor or a compromised developer account can read logs, they can escalate from "read-only access to logs" to "full read/write access to the storage cluster."

Furthermore, because this is an HMAC secret, it's likely a long-lived static key. Unless the organization has automated secret rotation (unlikely if they are using static HMAC keys), this leaked credential could remain valid for years.