CVE-2026-116691: Cleartext LLM Provider API Key Disclosure in MLflow CrewAI Autologging

MLflow is an open-source platform designed to manage the machine learning and artificial intelligence lifecycle, providing experiment tracking, model registry, and OpenTelemetry-compatible observability tracing. To instrument multi-agent applications built with the CrewAI framework, MLflow exposes the mlflow.crewai.autolog() module. This integration automatically hooks into key CrewAI lifecycle events—including agent execution, task execution, and crew kickoff—to capture inputs, outputs, execution duration, and component metadata as structured trace spans.

In MLflow versions starting from 2.19.0 and prior to 3.17.0, the CrewAI autologging module contains an information disclosure vulnerability classified under CWE-532 (Insertion of Sensitive Information into Log File), CWE-200 (Exposure of Sensitive Information to an Unauthorized Actor), and CWE-312 (Cleartext Storage of Sensitive Information). When a CrewAI Agent or Crew is initialized with an LLM instance that holds an explicit API key (for example, LLM(model="gpt-4.1-mini", api_key="sk-proj-...")), the autologging hooks extract the internal state of these objects by iterating over their __dict__ attributes without recursively scrubbing nested credential fields.

During span creation, the unsanitized LLM instances are converted into string representations or expanded as raw dictionaries. Because CrewAI's LLM class retains the api_key parameter as a standard instance attribute, calling str() on the object or dumping its dictionary embeds the raw secret directly into the resulting telemetry payload. These spans are then automatically exported over the network to the configured MLflow tracking server or downstream OpenTelemetry Protocol (OTLP) collector.

The exposure occurs transparently during normal agent execution and requires no faulty logging logic by the application developer. Any user, service, or monitoring platform with read access to the MLflow trace UI, trace REST API, backing storage database, or remote observability platform can retrieve the plaintext LLM provider credentials.

The root cause of CVE-2026-116691 is an incomplete sanitization architecture in mlflow/crewai/autolog.py that failed to account for object composition and nested Pydantic models within CrewAI components. While the developers recognized the need to redact secrets in isolated contexts, the sanitization logic was applied inconsistently across two distinct serialization code paths: span attribute extraction and function input construction.

The first leakage path occurs in _get_agent_attributes(instance), which is invoked by _set_span_attributes() whenever an Agent instance is traced. This function iterates over instance.__dict__.items() to collect configuration metadata for the span. In vulnerable versions, the loop applied specialized handling only to the tools key (via _parse_tools()) and the embedder key (via _sanitize_value()). All other attributes on the Agent instance—specifically llm, function_calling_llm, agent_executor, and crew—bypassed _sanitize_value() entirely and were coerced directly via str(value). When Python evaluated str(agent.llm), the default object representation of the CrewAI LLM instance serialized all internal fields, outputting a string containing api_key='sk-proj-...'.

Notably, MLflow's dedicated extractor for standalone LLM calls, _get_llm_attributes(instance), already contained an explicit filter (elif key in ["callbacks", "api_key"]: continue) to prevent logging the api_key attribute on direct LLM spans. However, because _get_agent_attributes() serialized the Agent object's sub-components via string conversion rather than delegating nested LLM objects to _get_llm_attributes() or a recursive sanitizer, the protection was completely bypassed during agent-level tracing.

The second independent leakage path resides in _construct_full_inputs(func, *args, **kwargs), which captures the arguments passed to traced CrewAI methods and records them under the mlflow.spanInputs attribute. This function constructed a dictionary by expanding any argument possessing a __dict__ attribute (k: v.__dict__ if hasattr(v, "__dict__") else v) without invoking _sanitize_value() on the resulting structure. When methods such as Task.execute_sync or Agent.execute_task were executed, the Agent instance was passed as a function argument, causing its internal __dict__—and the nested LLM object containing the plaintext api_key—to be serialized directly into the span inputs.

In vulnerable versions of MLflow (>= 2.19.0, < 3.17.0), the implementation of _get_agent_attributes(), _construct_full_inputs(), and _sanitize_value() in mlflow/crewai/autolog.py lacked recursive object traversal and comprehensive key matching. Below is the vulnerable code prior to the fix:

# Vulnerable implementation in mlflow/crewai/autolog.py
def _construct_full_inputs(func, *args, **kwargs):
    signature = inspect.signature(func)
    arguments = signature.bind_partial(*args, **kwargs).arguments
    if "self" in arguments:
        arguments.pop("self")
 
    # VULNERABILITY (Path 2): Expands object __dict__ without sanitization,
    # exposing nested LLM instances inside Agent arguments.
    return {
        k: v.__dict__ if hasattr(v, "__dict__") else v
        for k, v in arguments.items()
        if v is not None and _is_serializable(v)
    }
 
def _get_agent_attributes(instance):
    agent = {}
    for key, value in instance.__dict__.items():
        if key == "tools":
            value = _parse_tools(value)
        elif key == "embedder":
            value = _sanitize_value(value)
        if value is None:
            continue
        # VULNERABILITY (Path 1): Fields like 'llm', 'function_calling_llm', and
        # 'agent_executor' are not sanitized before calling str(value).
        agent[key] = str(value)
    return agent
 
def _sanitize_value(val):
    if val is None:
        return None
    sensitive_keys = ["api_key", "secret", "password", "token"]
    # VULNERABILITY: Only handles plain dicts and lists; ignores Pydantic models
    # and CrewAI LLM instances.
    if isinstance(val, dict):
        sanitized = {}
        for k, v in val.items():
            if any(sensitive in k.lower() for sensitive in sensitive_keys):
                continue
            sanitized[k] = _sanitize_value(v)
        return sanitized
    elif isinstance(val, list):
        return [_sanitize_value(item) for item in val]
    return val

Pull Request #25845 (commit 20927930d27f0072fcfeb7f024f24ad8c5e77c47) remediates the vulnerability by overhauling how span attributes, function inputs, and outputs are sanitized across mlflow/crewai/autolog.py. First, the patch introduces _SENSITIVE_KEY_PATTERN, a compiled regular expression that matches credential-related key names (api_key, secret, password, authorization, credential, and standalone or camelCase token/Token suffixes) while preserving benign configuration keys such as max_tokens, tokenizer, and token_usage.

# Patched implementation in mlflow/crewai/autolog.py
_SENSITIVE_KEY_PATTERN = re.compile(
    r"(?i:api_?key|secret|password|authorization|credential)"  # anywhere in the key
    r"|(?<![A-Za-z])(?i:token)$"  # snake_case / standalone `token`
    r"|(?<=[a-z])Token$"  # camelCase `accessToken`
)
 
def _set_sanitized_attribute(span: LiveSpan, key: str, value: Any):
    # Drop top-level attributes whose key name matches the sensitive pattern
    if _SENSITIVE_KEY_PATTERN.search(key):
        return
    value = _sanitize_value(value)
    span.set_attribute(key, str(value) if isinstance(value, list) else value)
 
def _get_agent_attributes(instance):
    agent = {}
    for key, value in instance.__dict__.items():
        if _SENSITIVE_KEY_PATTERN.search(key):
            continue
        if key == "tools":
            value = _parse_tools(value)
        # FIX: Sanitize all values (including nested LLM objects) BEFORE stringifying
        value = _sanitize_value(value)
        if value is None:
            continue
        agent[key] = str(value)
    return agent

Second, the patch replaces the naive _sanitize_value() implementation with a cycle-aware, fail-closed recursive sanitizer (_sanitize_value_impl). When encountering a CrewAI LLM instance (detected via _is_crewai_llm()), the sanitizer reduces the entire object to its model identifier string via _get_llm_summary(), preventing both str(llm) and dictionary dumps from exposing internal credentials. For other pydantic.BaseModel subclasses (such as Agent, Task, and Crew), the sanitizer recursively traverses val.__dict__ while tracking object memory addresses (id(val)) in an ancestors set to prevent infinite recursion on circular references (such as Agent.crew -> Crew.agents -> Agent).

def _sanitize_value(val):
    try:
        return _sanitize_value_impl(val, set())
    except Exception as e:
        _logger.debug("Failed to sanitize %s for span logging: %s", type(val).__name__, e)
        # Fail-closed: return placeholder on error rather than logging unsanitized data
        return f"<{type(val).__name__}: omitted, sanitization failed>"
 
def _sanitize_value_impl(val, ancestors: set[int]):
    if val is None or isinstance(val, (str, int, float, bool)):
        return val
    if id(val) in ancestors:
        return type(val).__name__
    ancestors.add(id(val))
    try:
        if isinstance(val, dict):
            return {
                k: _sanitize_value_impl(v, ancestors)
                for k, v in val.items()
                if not (isinstance(k, str) and _SENSITIVE_KEY_PATTERN.search(k))
            }
        if isinstance(val, (list, tuple, set)):
            sanitized = [_sanitize_value_impl(item, ancestors) for item in val]
            return tuple(sanitized) if isinstance(val, tuple) else sanitized
        if _is_crewai_llm(val):
            return _get_llm_summary(val)
        if isinstance(val, pydantic.BaseModel):
            return _sanitize_value_impl(val.__dict__, ancestors)
        return val
    finally:
        ancestors.discard(id(val))

The fix also wraps the return dictionary of _construct_full_inputs() and the outputs of patched_standalone_call() and patched_class_call() in _sanitize_value(). The remediation is comprehensive for known dictionary keys, Pydantic models, and CrewAI BaseLLM subclasses. However, custom non-Pydantic Python classes that do not inherit from pydantic.BaseModel or BaseLLM and are not expanded into dict structures will fall through to return val in _sanitize_value_impl(), which could still allow sensitive fields in custom third-party wrapper objects to be stringified if attached directly to an Agent.

Exploiting or triggering this vulnerability requires no specialized attacker action during execution; the credential leakage occurs automatically whenever an application running a vulnerable MLflow version (>= 2.19.0, < 3.17.0) enables mlflow.crewai.autolog() and instantiates a CrewAI LLM with an api_key argument. An adversary with low-privileged read access (PR:L) to the MLflow tracking server, artifact store, or connected OTLP telemetry backend can subsequently harvest the exposed keys over the network (AV:N).

During a single standard crew.kickoff() execution, a single API key is duplicated across five distinct attribute locations spanning two separate OpenTelemetry spans:

1. Agent Span (e.g., Support Responder):
   • attributes.llm
   • attributes.function_calling_llm
   • attributes.agent_executor
   • attributes.mlflow.spanInputs
2. Task Span (e.g., Task.execute_sync):
   • attributes.mlflow.spanInputs

PoC 1: In-Memory Attribute Extraction

The following proof-of-concept demonstrates the underlying serialization flaw directly in memory without requiring an active network connection, external LLM API call, or running MLflow server:

import importlib
from crewai import LLM, Agent
 
# Load the internal MLflow CrewAI autologging module
autolog = importlib.import_module("mlflow.crewai.autolog")
 
# Initialize a CrewAI LLM instance with an explicit API key
llm = LLM(
    model="gpt-4.1-mini",
    api_key="sk-proj-EXAMPLE1234567890",
)
 
agent = Agent(
    role="r",
    goal="g",
    backstory="b",
    llm=llm,
)
 
# Invoke the vulnerable attribute extraction routine
attrs = autolog._get_agent_attributes(agent)
 
# Verify that the plaintext API key is embedded in the serialized 'llm' attribute
assert "sk-proj-EXAMPLE1234567890" in attrs["llm"]
print("Leaked attribute string:")
print(attrs["llm"])

Executing this script on MLflow < 3.17.0 produces output matching the following structure:

Leaked attribute string:
llm_type='openai' model='gpt-4.1-mini' ... api_key='sk-proj-EXAMPLE1234567890' ...

PoC 2: End-to-End OTLP Telemetry Exfiltration

To observe the credential leak across the network via MLflow's OpenTelemetry exporter, configure the environment to point to a local OTLP HTTP receiver:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export CREWAI_DISABLE_TELEMETRY=true

Next, execute a minimal CrewAI workflow with MLflow autologging enabled:

import mlflow
from crewai import Agent, Crew, LLM, Task
 
mlflow.crewai.autolog()
 
llm = LLM(
    model="gpt-4.1-mini",
    api_key="sk-proj-EXAMPLE1234567890",
)
 
agent = Agent(
    role="Support Responder",
    goal="Answer support questions",
    backstory="A support agent",
    llm=llm,
)
 
task = Task(
    description="Answer the user's question",
    expected_output="A response",
    agent=agent,
)
 
crew = Crew(
    agents=[agent],
    tasks=[task],
)
 
try:
    crew.kickoff()
except Exception:
    # Even when the LLM API call fails due to a dummy key, spans are emitted
    pass

Even though crew.kickoff() raises an authentication exception against the upstream LLM provider when using a synthetic key, MLflow finalizes and flushes the trace spans prior to exit. Inspecting the incoming protobuf ExportTraceServiceRequest packets on port 4318 reveals sk-proj-EXAMPLE1234567890 transmitted in cleartext within both the span attributes and mlflow.spanInputs.

CVE-2026-116691 carries a CVSS v3.1 base score of 6.5 (Medium) with the vector CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N. The score reflects high confidentiality impact (C:H) combined with low attack complexity (AC:L) and network accessibility (AV:N), gated only by the requirement for low privileges (PR:L) to read trace data from the tracking server or telemetry platform.

In enterprise machine learning environments, MLflow tracking servers and observability backends (such as Datadog, Honeycomb, Jaeger, or Grafana Tempo) are routinely shared across wide engineering, data science, and operations teams. Developers and analysts who have read-only access to experiment traces for debugging or performance evaluation are frequently not authorized to access production LLM provider credentials. Because the API keys are persisted in plaintext inside trace span metadata, any user with trace viewing permissions can extract production secrets for OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, or other configured providers.

Once an attacker obtains a valid LLM provider API key from the trace logs, they can authenticate directly to the external AI provider API outside the boundary of the MLflow environment. Depending on the scope of the compromised credential, this enables unauthorized consumption of paid inference quotas, access to fine-tuned proprietary models, or retrieval of sensitive organization data stored within provider-managed vector stores, files, and assistants.

> [!NOTE] > While the CVSS vector assigns S:U (Scope Unchanged) and I:N/A:N with respect to the MLflow application itself, the downstream consequence of leaked cloud AI credentials extends to third-party API billing abuse and potential data exfiltration from the upstream LLM provider account.

Patching and Workarounds

To remediate CVE-2026-116691, upgrade the mlflow package to version 3.17.0 or later, which includes commit 20927930d27f0072fcfeb7f024f24ad8c5e77c47:

pip install --upgrade "mlflow>=3.17.0"

If immediate upgrading is blocked by dependency constraints, apply one of the following workarounds:

1. Disable CrewAI Autologging: Explicitly disable the CrewAI instrumentation in application code by calling mlflow.crewai.autolog(disable=True).
2. Use Environment Variables for Credentials: Avoid passing the api_key argument directly to the LLM(...) constructor in Python code. Instead, rely on standard provider environment variables (such as OPENAI_API_KEY or ANTHROPIC_API_KEY) read implicitly by the underlying SDKs at request time, ensuring the key is not stored directly on the LLM instance's __dict__.

Post-Upgrade Verification

Platform engineers can verify that an environment is no longer vulnerable by executing the following verification check:

import importlib
from crewai import LLM, Agent
 
autolog = importlib.import_module("mlflow.crewai.autolog")
llm = LLM(model="gpt-4o-mini", api_key="sk-test-SECRETKEY12345")
agent = Agent(role="test", goal="test", backstory="test", llm=llm)
 
attrs = autolog._get_agent_attributes(agent)
if "sk-test-SECRETKEY12345" in str(attrs):
    print("[-] VULNERABLE: Cleartext API key found in agent attributes.")
else:
    print("[+] PATCHED: Credentials sanitized from agent attributes.")

Threat Hunting and Incident Response

Upgrading MLflow prevents future leaks but does not retroactively sanitize historical traces stored in backend databases or external observability platforms. Security teams must conduct retrospective threat hunting across all trace repositories that ingested data from MLflow versions 2.19.0 through 3.16.x:

1. Audit Stored Spans: Query MLflow backend storage (SQL databases, artifact buckets) and OTLP collectors for spans generated by CrewAI containing the attributes attributes.llm, attributes.function_calling_llm, attributes.agent_executor, and attributes.mlflow.spanInputs.
2. Scan for Credential Patterns: Apply regular expressions against stored span JSON payloads to identify exposed keys:
   • OpenAI Project Keys: sk-proj-[A-Za-z0-9_-]+
   • Standard OpenAI Keys: sk-[A-Za-z0-9]{32,}
   • Stringified Key Assignments: api_key=['"][^'"]+['"]
   • Bearer Tokens: Bearer\s+[A-Za-z0-9\._\-]+
3. Rotate and Purge: Immediately revoke and rotate any API key that was passed to a CrewAI LLM object while mlflow.crewai.autolog() was active, and purge or redact the affected trace records from all storage backends.