CVE-2026-84366: Plaintext AWS Credential Exposure in Scrapy S3DownloadHandler

Scrapy is an open-source web crawling and scraping framework for Python widely used to extract structured data from websites. Among its capabilities, Scrapy supports downloading objects directly from Amazon Simple Storage Service (S3) using custom s3:// URIs. This functionality is implemented in the Scrapy core downloader via the S3DownloadHandler class.\n\nPrior to Scrapy version 2.17.0, a vulnerability existed in this handler where S3 API requests were routed over plaintext HTTP rather than secure HTTPS. The root of this behavior was a default configuration logic that initialized target connections with the unencrypted HTTP scheme unless explicitly configured otherwise.\n\nBecause S3 requests typically require authentication headers when accessing private buckets, the S3DownloadHandler signed these requests prior to dispatch. Consequently, AWS Signature Version 4 parameters, access keys, and session tokens were transmitted over the network in cleartext, presenting an exposure risk.

The core issue resides in the scheme selection logic of the S3DownloadHandler.download_request method in scrapy/core/downloader/handlers/s3.py. When a scraper processes an s3:// URL, the handler must translate it into an HTTP or HTTPS API endpoint URL that can be requested via Scrapy's underlying download machinery.\n\nTo determine which protocol to use, the handler evaluated the boolean expression request.meta.get("is_secure"). In Python, retrieving a non-existent key from a dictionary using .get(key) returns None by default. Because the is_secure key was not present in the request meta dictionary during standard crawling operations, this expression evaluated to None.\n\nIn Python's conditional evaluation, None is falsy. Therefore, the ternary operation defaulted to selecting the 'http' scheme. This resulted in all subsequent outbound API calls, including those carrying signature headers, being constructed as http://<bucket>.s3.amazonaws.com connections, exposing sensitive cryptographic material.

To understand the vulnerability and its remediation, we examine the codebase of the S3 download handler before and after the fix.\n\nIn the vulnerable version, the scheme was assigned using a falsy check on the optional metadata parameter:\n\npython\n# Vulnerable Implementation\ndef download_request(self, request: Request) -> Response:\n p = urlparse_cached(request)\n # request.meta.get("is_secure") returns None, which evaluates to False\n scheme = "https" if request.meta.get("is_secure") else "http"\n bucket = p.hostname\n path = p.path + "?" + p.query if p.query else p.path\n url = f"{scheme}://{bucket}.s3.amazonaws.com{path}"\n\n\nThe patch implemented in version 2.17.0 corrects this logical path by explicitly checking for a boolean False value, thereby making HTTPS the default choice for all requests unless an explicit opt-out is supplied:\n\npython\n# Patched Implementation\ndef download_request(self, request: Request) -> Response:\n p = urlparse_cached(request)\n # Explicit check for False ensures None defaults to "https"\n scheme = "http" if request.meta.get("is_secure") is False else "https"\n bucket = p.hostname\n path = p.path + "?" + p.query if p.query else p.path\n url = f"{scheme}://{bucket}.s3.amazonaws.com{path}"\n\n\nThis adjustment guarantees that standard requests without the is_secure metadata key default to secure HTTPS connections while maintaining compatibility for local testing frameworks (like LocalStack) that operate over HTTP.

An attacker positioned on the network path between the crawler host and the AWS S3 endpoint can intercept the plaintext HTTP communications. This position can be achieved through ARP spoofing, DNS hijacking, or by operating on compromised intermediate routing infrastructure.\n\nOnce positioned, the attacker can sniff port 80 traffic for requests targeting *.s3.amazonaws.com. These requests contain standard AWS headers such as Authorization (containing credential scope and AWS Access Key ID) and X-Amz-Security-Token (containing temporary session tokens when running under IAM roles).\n\nAdditionally, because the connection is unencrypted, an active adversary can perform a Man-in-the-Middle (MITM) attack. They can modify the S3 payload in transit, inject redirect responses (such as HTTP 301/302), or alter the scraped content to feed poisoned data to downstream processing engines.\n\nmermaid\ngraph LR\n A["Scrapy Crawler"] -->|"Plaintext HTTP request\nwith AWS Signature"| B["Adversary / MITM Sniffer"]\n B -->|"Captured Credentials"| C["Unauthorized AWS Access"]\n B -->|"Forwarded/Modified Payload"| D["Target S3 Bucket"]\n

The exposure of AWS credentials and session tokens represents a high-severity threat to cloud infrastructure. An attacker who successfully extracts these credentials can gain unauthorized access to the underlying S3 buckets, allowing them to read, write, or delete sensitive data depending on the permissions associated with the compromised credentials.\n\nIf the crawler runs with a high-privilege IAM role or standard long-lived AWS Access Keys, the blast radius extends beyond the targeted bucket. This could lead to broader cloud environment compromise, data exfiltration, or resource hijacking for cryptocurrency mining.\n\nFurthermore, the lack of integrity protection over HTTP enables active data tampering. The results of the web scraping operations cannot be trusted, as an intermediate router could inject false data, presenting significant integrity issues for applications relying on the crawled dataset.

The primary remediation is upgrading to Scrapy 2.17.0 or later, which shifts the default protocol to HTTPS.\n\nFor environments where upgrading is not immediately feasible, developers must secure their S3 requests manually. This can be accomplished by explicitly defining the is_secure parameter in the request metadata:\n\npython\n# Workaround for Scrapy < 2.17.0\nyield scrapy.Request(\n url="s3://my-bucket/target-file.json",\n meta={"is_secure": True}\n)\n\n\nAdditionally, security teams should implement SCP (Service Control Policies) or IAM policies that enforce secure transport (HTTPS) for all S3 API interactions, causing any unencrypted S3 requests to be rejected at the AWS API gateway level.