CVE-2026-88982: Cross-Tenant Execution Cache Poisoning and Output Substitution in Kubeflow Pipelines

Kubeflow Pipelines (KFP) provides a platform for building and deploying portable, scalable machine learning workflows on Kubernetes. To optimize resource utilization and reduce redundant computation, KFP includes an execution caching subsystem (cache-server). For legacy V1 pipelines and raw Argo Workflows, this subsystem operates as a Kubernetes mutating admission webhook coupled with a persistent database backend and a pod completion watcher.

When a pipeline step is submitted for execution, the Kubernetes API server forwards the pod creation request to the cache-server mutating admission webhook. The webhook inspects the Argo workflow template embedded within the pod annotations, computes a deterministic SHA-256 hash representing the step's execution specification, and queries the backing database for prior successful executions matching that hash. If a valid, non-expired cache entry exists, the webhook mutates the pod specification to bypass actual container execution and injects the precomputed output artifacts directly into the pod metadata.

In Kubeflow Pipelines versions prior to 2.18.0, this caching architecture lacked tenant isolation across three distinct layers: key derivation, database storage, and completion verification. In multi-user, multi-tenant clusters where distinct Kubernetes namespaces share a single cache-server instance, the absence of namespace binding created a global, unpartitioned cache namespace.

This architectural defect maps to CWE-639 (Authorization Bypass Through User-Controlled Key) and CWE-345 (Insufficient Verification of Data Authenticity). An authenticated tenant with privileges to create pods or submit workflows within their designated namespace can exploit this shared state to read sensitive step outputs produced by other tenants or inject fabricated outputs into downstream pipeline steps running in victim namespaces.

The root cause of CVE-2026-88982 is the omission of the Kubernetes namespace from the execution cache identity model, compounded by improper trust placed in client-modifiable pod metadata during the cache population phase. The vulnerability manifests across three specific mechanisms within the backend/src/cache package.

First, during the admission phase in backend/src/cache/server/mutation.go, the function generateCacheKeyFromTemplate derived the executionHashKey exclusively by filtering the workflows.argoproj.io/template annotation against a structural skeleton (templateSkeleton) and hashing the marshaled JSON structure. The admission review request's namespace (req.Namespace) was ignored. Consequently, any two pods across the entire cluster sharing identical container images, command arguments, and input parameters produced the exact same 64-character hexadecimal SHA-256 cache key, regardless of which tenant namespace initiated the workload.

Second, the relational database model defined in backend/src/cache/model/execution_cache.go and queried via backend/src/cache/storage/execution_cache_store.go contained no column or index for tenant or namespace attribution. Database lookups executed by GetExecutionCache filtered records solely where ExecutionCacheKey matched the computed template hash. The storage layer treated all cached execution records as globally accessible across all namespaces connected to the webhook.

Third, the cache population mechanism in backend/src/cache/server/watcher.go suffered from insufficient verification of data authenticity. When a non-cached pod completed execution, the WatchPods controller extracted the cache key directly from the pod's mutable annotation pipelines.kubeflow.org/execution_cache_key rather than recomputing the hash from the authoritative pod specification and namespace. Because users with pod creation or update privileges in a tenant namespace can set arbitrary annotations on their own pods, an attacker could explicitly assign a target victim's execution_cache_key to a malicious pod. Upon pod termination, the watcher blindly persisted the attacker's output artifacts under the specified key.

The remediation implemented in commit 620d0cc89ae0d042a484a003f8ffe6e9a55d347e (PR #14359) addresses all three failure points by enforcing strict namespace binding throughout the cache lifecycle. Examining the code diffs illustrates the transition from a namespace-blind architecture to an isolated multi-tenant model.

In backend/src/cache/server/mutation.go, the original key generation function accepted only the serialized Argo template string. The patched version requires the pod's namespace and embeds both the namespace and the filtered template map into an outer structure prior to hashing:

// BEFORE (Vulnerable - backend/src/cache/server/mutation.go)
func generateCacheKeyFromTemplate(template string) (string, error) {
    // ... template parsing and filtering ...
    cacheKeyMap := intersectStructureWithSkeleton(templateMap, templateSkeleton)
    b, err = json.Marshal(cacheKeyMap)
    if err != nil {
        return "", err
    }
    hash := sha256.New()
    hash.Write(b)
    return hex.EncodeToString(hash.Sum(nil)), nil
}
 
// AFTER (Patched - backend/src/cache/server/mutation.go)
func generateCacheKeyFromTemplate(template string, namespace string) (string, error) {
    if namespace == "" {
        return "", fmt.Errorf("cache key requires a pod namespace")
    }
    cacheKeyMap, err := filterTemplateForCacheKey(template)
    if err != nil {
        return "", err
    }
    // Binds the cryptographic hash directly to the tenant namespace
    return hashCacheKey(map[string]interface{}{
        "namespace": namespace,
        "template":  cacheKeyMap,
    })
}

Furthermore, inside the MutatePodIfCached admission handler, the patch ensures that the namespace used for key derivation and database querying is sourced authoritatively from the Kubernetes API server's AdmissionRequest context (req.Namespace), preventing a caller from spoofing the namespace inside the raw pod object metadata:

// Patched validation in MutatePodIfCached (backend/src/cache/server/mutation.go)
if req.Namespace == "" || (pod.Namespace != "" && pod.Namespace != req.Namespace) {
    log.Printf("Skipping cache lookup for pod %s: missing or inconsistent namespace", pod.Name)
    return nil, nil
}

In backend/src/cache/server/watcher.go, the completion watcher was refactored to eliminate blind trust in the pipelines.kubeflow.org/execution_cache_key annotation. When a pod finishes execution, the watcher extracts the Argo template and the pod's immutable Kubernetes namespace, recomputes the expected cache key, and verifies that it matches the annotation before writing to the database:

// AFTER (Patched - backend/src/cache/server/watcher.go)
executionTemplate, exists := getArgoTemplate(pod)
if !exists {
    return nil
}
// Authoritatively recompute the cache key using the pod's actual namespace
executionKey, err := generateCacheKeyFromTemplate(executionTemplate, pod.Namespace)
if err != nil {
    return fmt.Errorf("cannot determine cache identity for pod %s/%s: %w", pod.Namespace, pod.Name, err)
}
// Verify the mutable annotation matches the recomputed identity
if pod.Annotations[ExecutionKey] != executionKey {
    return fmt.Errorf("skipping cache write for pod %s/%s: execution cache key does not match namespace and template", pod.Namespace, pod.Name)
}

Finally, the storage schema in backend/src/cache/model/execution_cache.go was updated to include a Namespace column indexed alongside ExecutionCacheKey via a composite index (idx_cache_namespace_key). All database reads and writes in execution_cache_store.go now explicitly scope queries by both Namespace and ExecutionCacheKey.

> [!NOTE] > Fix Completeness Assessment: The patch comprehensively resolves the cross-tenant vulnerability within the webhook and watcher components. However, administrators must note the introduction of the ALLOW_LEGACY_CACHE_FALLBACK environment variable. If enabled during upgrades, the store falls back to querying legacy rows where Namespace = '' using a template-only hash. While legacy rows cannot be newly poisoned once all replicas run 2.18.0, enabling this flag allows cross-tenant reads of historical pre-upgrade cache entries. The flag defaults to false, which maintains strict isolation.

Exploiting CVE-2026-88982 requires an authenticated user with permissions to submit Kubeflow V1 pipelines, raw Argo workflows, or standard pods within at least one tenant namespace in a shared cluster. No elevated cluster-admin privileges or network interception capabilities are required. Exploitation follows two primary methodologies: direct annotation spoofing (cache poisoning) and template replication (output substitution and data exfiltration).

Method 1: Targeted Cache Poisoning via Annotation Spoofing

Because the pre-patch WatchPods controller trusted pod annotations upon pod completion, an attacker does not even need to execute the exact container image used by the victim to poison the cache if the target execution_cache_key can be pre-calculated.

1. Reconnaissance & Hash Calculation: The attacker identifies a common pipeline step used by a target tenant (for example, a standard data validation component or model evaluation script from a shared internal registry). Using the known container image tag, command array, and input parameters, the attacker locally computes the SHA-256 hash using the intersectStructureWithSkeleton logic.
2. Pod Submission: In the attacker's own namespace (tenant-attacker), the attacker deploys a lightweight pod configured with the following metadata annotations:
   • pipelines.kubeflow.org/execution_cache_key: Set to the calculated target SHA-256 hash.
   • workflows.argoproj.io/outputs: Set to a JSON payload containing fabricated output parameters or artifact references pointing to attacker-controlled storage.
   • workflows.argoproj.io/template: Set to a valid Argo template JSON string.
3. Cache Persistence: When the attacker's pod terminates with a Succeeded phase, the cluster-wide cache-server watcher detects the completion event, reads the spoofed execution_cache_key annotation, and inserts the fabricated workflows.argoproj.io/outputs into the database.
4. Victim Compromise: When the victim namespace (tenant-victim) subsequently runs its pipeline, the mutating admission webhook intercepts the victim pod creation, computes the identical SHA-256 key, retrieves the attacker's database record, replaces the victim's container execution with a no-op command (echo), and injects the poisoned outputs into the victim workflow.

Method 2: Cross-Tenant Output Exfiltration

To read sensitive cached outputs produced by another tenant, the attacker leverages the admission webhook's lookup behavior.

1. Template Replication: The attacker constructs a workflow step in tenant-attacker whose filtered Argo template fields (container, inputs, outputs, script) exactly match a step previously executed by tenant-victim within the victim's configured MaxCacheStaleness window.
2. Webhook Mutation: Upon submitting the pod, the mutating admission webhook calculates the namespace-blind hash and queries the database.
3. Data Extraction: Finding the record created by tenant-victim, the webhook mutates the attacker's pod and writes the victim's cached output parameters directly into the attacker pod's workflows.argoproj.io/outputs annotation. The attacker then reads their own pod specification via kubectl get pod -o yaml to extract the victim's data.

CVE-2026-88982 carries a CVSS v3.1 base score of 5.4 (Medium) with the vector CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N. While the vulnerability does not grant direct remote code execution on the underlying Kubernetes nodes or compromise the Kubernetes API server directly, its impact on machine learning integrity and tenant confidentiality in multi-user platforms is substantial.

Integrity Impact (Low/Medium): By substituting execution outputs across tenant boundaries, an attacker can corrupt downstream pipeline stages in victim namespaces. In machine learning workflows, step outputs frequently dictate model promotion gates, hyperparameter selection, evaluation metrics, or dataset URI pointers. Injecting falsified validation metrics can force a victim pipeline to promote an underperforming or backdoored model to production. Furthermore, if a downstream step consumes a cached output parameter as a storage URI or script argument without secondary sanitization, output substitution can pivot into arbitrary data ingestion or code execution within the victim's pipeline context.

Confidentiality Impact (Low): Any scalar parameters, metrics, or artifact metadata recorded in the workflows.argoproj.io/outputs annotation by a cached V1 step are exposed to any other tenant capable of predicting or replicating the step's template specification. In enterprise environments where tenants share base component definitions and run jobs over predictable parameter grids, reconstructing a target's template structure is trivial.

Scope Constraints: The impact is bounded by two technical factors. First, Kubeflow Pipelines V2 component executions utilize a separate metadata and caching architecture (MLMD) and bypass the legacy mutating webhook entirely (pipelines.kubeflow.org/v2_component: "true"). Second, the vulnerability only affects multi-tenant deployments where multiple untrusted or semi-trusted users share a single KFP installation; single-tenant clusters do not cross trust boundaries during cache hits.

Organizations operating multi-tenant Kubeflow Pipelines deployments using V1 pipelines or raw Argo caching must upgrade to Kubeflow Pipelines version 2.18.0 or later. Because the remediation modifies both the cryptographic key derivation formula and the underlying relational database schema, administrators must follow a specific operational sequence during the upgrade to prevent race conditions.

Upgrade Procedure

1. Scale Down Existing Replicas: Before deploying the 2.18.0 cache-server image, scale all existing pre-2.18.0 cache-server deployment replicas down to zero. Running mixed versions during a rolling update allows legacy replicas to continue writing unscoped records (Namespace = '') and serving unverified cache hits.
2. Deploy Patched Version & Run Migrations: Deploy the 2.18.0 cache-server. Upon initialization, the server executes GORM AutoMigrate, adding the Namespace column (VARCHAR 63, default '') and building the composite index idx_cache_namespace_key on (Namespace, ExecutionCacheKey). Wait for the database migration to complete before submitting new pipeline workloads.
3. Maintain Strict Isolation Configuration: Ensure the environment variable ALLOW_LEGACY_CACHE_FALLBACK remains unset or explicitly set to false in the cache-server deployment manifest. While setting ALLOW_LEGACY_CACHE_FALLBACK=true preserves warm-cache hits against pre-upgrade database entries, those legacy entries lack tenant provenance and remain susceptible to cross-tenant reads.

Interim Workarounds

If an immediate upgrade to 2.18.0 is not operationally feasible, administrators can neutralize the vulnerability by disabling the execution cache webhook or disabling caching at the workflow level:

• Cluster-Wide Webhook Deactivation: Delete or disable the MutatingWebhookConfiguration responsible for routing pod creation requests to the cache-server service, and scale down the cache-server deployment. All pipeline steps will execute normally without caching.
• Migrate to KFP V2: Transition pipeline definitions to the Kubeflow Pipelines V2 SDK/IR spec. Pods carrying the pipelines.kubeflow.org/v2_component: "true" label are ignored by both the legacy mutating webhook and the pod watcher.

> [!WARNING] > Post-Upgrade Operational Behavior: Because pre-existing cache rows have an empty Namespace value and new cache keys incorporate the namespace into the SHA-256 hash, all existing cache entries are effectively invalidated upon upgrading to 2.18.0. Administrators should anticipate a temporary spike in cluster compute utilization as pipeline steps re-execute to populate the namespace-scoped cache.