Groundhog Day in the Data Lake: Replaying lakeFS Requests Forever (CVE-2025-68671)

lakeFS is an excellent tool for data engineers. It treats your object storage (S3, Azure Blob, GCS) like a Git repository, allowing you to branch, commit, and merge terabytes of data. To make integration easy, it exposes an S3-compatible gateway. This means you can point your standard AWS CLI, boto3 scripts, or Spark jobs at lakeFS, and they just work. It's magic.

But here is the thing about emulating S3: you have to actually emulate the security protocols, not just the API endpoints. AWS Signature Version 4 (SigV4) is a beast of a protocol involving canonicalization, HMAC-SHA256 hashing, and scoped signing keys. It is designed to be robust. However, cryptography is brittle. You can get the complex math perfectly right—hashing the payload, signing the headers—but if you miss one logical check, the whole castle collapses.

In CVE-2025-68671, the lakeFS team got the math right but failed the history class. They verified that the signature matched the secret key, but they forgot to check when the signature was generated. In the world of distributed systems, if you don't check the clock, you are doomed to repeat the past—literally.

Let's talk about Replay Attacks. In a standard authentication protocol, we prevent bad actors from recording a valid request and sending it again later by using two main mechanisms: Nonces (number used once) or Timestamps.

AWS SigV4 relies heavily on timestamps. When a client signs a request, they include an X-Amz-Date header (or query parameter). The signature is calculated over this timestamp. The server is supposed to do two things:

1. Verify the signature is mathematically valid for that timestamp.
2. Verify the timestamp is current (usually within ±15 minutes of the server's clock).

If the server skips step #2, the signature becomes a "Golden Ticket." It doesn't matter if the request was generated five minutes ago or five months ago. As long as the underlying Access Key Secret hasn't been rotated, that request remains valid forever.

Prior to version 1.75.0, lakeFS was essentially acting like a bouncer checking ID cards for a photo match but ignoring the "Expires: 2012" printed in bold red letters. An attacker didn't need the private key; they just needed to listen on the wire or read a verbose log file once.

The vulnerability lived in pkg/gateway/sig/v4.go. The authentication logic was focused entirely on cryptographic verification and missed the temporal check. The fix, introduced in commit 92966ae611d7f1a2bbe7fd56f9568c975aab2bd8, adds the missing logic. It is a textbook example of "security checks we assume are there but aren't."

Here is the logic that was added. Notice the check for AmzMaxClockSkew (15 minutes) and the specific handling for Presigned URLs (which have their own expiration logic).

func (ctx *verificationCtx) verifyExpiration() error {
    // If it's a standard header-based auth
    if !ctx.AuthValue.IsPresigned {
        requestTime, _ := time.Parse(v4timeFormat, amzDate)
        now := time.Now().UTC()
        timeDiff := now.Sub(requestTime)
        
        // THE FIX: Enforce the 15-minute window
        if timeDiff.Abs() > AmzMaxClockSkew {
            return errors.ErrRequestTimeTooSkewed
        }
        return nil
    }
 
    // If it's a presigned URL
    expirationTime := requestTime.Add(time.Duration(ctx.AuthValue.Expires) * time.Second)
    if now.After(expirationTime) {
        return errors.ErrExpiredPresignRequest
    }
    return nil
}

Before this function existed, the code would simply return nil (success) as long as hmac.Equal(signature, calculatedSignature) was true. This diff effectively closes the time window from "Infinity" down to "15 minutes."

Exploiting this is trivially easy if you have visibility into the network or logs. Imagine a scenario where a CI/CD pipeline uploads a build artifact to lakeFS every night.

The Setup:

1. The CI server sends a PUT /repo/main/builds/latest.jar request.
2. The request includes the Authorization header with the SigV4 signature.
3. The attacker (Eve) is lurking on the network, or perhaps has read-access to the CI server's verbose output logs.

The Attack: Eve copies the full HTTP request. Three weeks later, the developers release a new version. latest.jar is updated. Eve, wanting to cause chaos, simply pipes the old request into netcat or curl.

# Eve replays the captured request from 3 weeks ago
curl -X PUT "https://lakefs.corp.local/repo/main/builds/latest.jar" \
  -H "X-Amz-Date: 20250101T000000Z" \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=AKIA.../20250101/us-east-1/s3/aws4_request, ... Signature=..." \
  --data-binary @malicious_old_version.jar

The Result: lakeFS receives the request. It checks the signature against the date 20250101. The signature matches. It then should check if 20250101 is close to Today. It doesn't. The write is accepted. The production build is silently downgraded to the vulnerable version from three weeks ago.

While a CVSS of 6.5 might look "Medium," the operational impact here is nasty. This is a Capture-Replay vulnerability (CWE-294), and in the context of a data versioning system, it enables "Time Travel Attacks."

1. Data Integrity Loss: An attacker can revert datasets to previous states without generating new commit IDs that might alert administrators. They are effectively unstitching the timeline of your data.
2. Denial of Service (DoS): If an attacker captures a DELETE request, they can periodically replay it. Every time you try to upload that file again, the attacker's script runs the DELETE replay, and the file vanishes. Troubleshooting this is a nightmare because the logs show a valid, authenticated user deleting the file.
3. Presigned URL Bypass: Presigned URLs are often used to give temporary access (e.g., "download this file for the next 5 minutes"). With this bug, a 5-minute link becomes a permanent link.

The mitigation is straightforward: Upgrade.

Remediation Steps:

1. Upgrade lakeFS: Move to version 1.75.0 or later immediately. This introduces the verifyExpiration logic.
2. Rotate Credentials: This is crucial. Patching the code prevents future invalid timestamps, but if an attacker has already captured a signed request, they might try to replay it. While the patch stops the replay, rotating the Access Keys invalidates the signature mathematically, ensuring total closure.

For developers building their own S3-compatible tools: Always validate X-Amz-Date. It is not metadata; it is a security control. If the time difference is greater than your skew tolerance (15 minutes is standard), drop the packet.