Peek-a-Boo: Unmasking Secrets in TeamCity Build Chains

CI/CD pipelines are the modern equivalent of a confessional booth. We whisper our deepest, darkest secrets—AWS keys, database credentials, signing certificates—into them, trusting that the priest (in this case, the build server) will keep them safe. We rely on that comforting row of asterisks (******) in the console output to tell us that our secrets are being handled with the discretion of a Swiss banker.

JetBrains TeamCity is a behemoth in this space. It’s the engine room for thousands of enterprises. One of its core promises is that if you define a parameter as type password, it stays that way. It gets scrubbed from the UI, masked in the logs, and treated like radioactive material. But CVE-2023-34223 proves that even the best secret keepers get chatty when things get complicated.

This isn't a buffer overflow or a complex heap grooming attack. It’s a logic failure in how secrets travel. It turns out that while TeamCity was great at keeping a secret within a single room, it shouted it out loud the moment it had to pass that secret through the hallway to the next room. This vulnerability effectively unmasks the credentials that act as the keys to your entire infrastructure kingdom.

To understand this bug, you have to understand TeamCity's dependency model. Complex software delivery isn't just one script; it's a chain. Build A compiles the code, Build B runs the tests, and Build C deploys the artifact. TeamCity handles this via Snapshot Dependencies. A downstream build (Build B) can inherit properties from an upstream build (Build A).

The syntax for this is straightforward: %dep.BuildA.ParameterName%. When TeamCity orchestrates this, it resolves the variable from the upstream build context and injects it into the downstream execution context. The vulnerability, tracked internally as TW-81338, resides in this handover.

The masking subsystem—the code responsible for scanning output streams and replacing sensitive strings with ******—relies on metadata. It needs to know that a specific string is supposed to be a secret. In versions prior to 2023.05, when a password parameter was pulled across a dependency boundary, that metadata was seemingly lost or ignored during the logging phase. The value was resolved to its plain text form before the logger realized it was a 'password' type, resulting in your production database password sitting naked in teamcity-agent.log.

While we don't have the raw Java patch diffs, we can reconstruct the failure accurately through the behavior of the build agent's variable resolution logic. In a secure implementation, a parameter defined as password (let's call it secure_param) is wrapped in a SecureString object or equivalent abstraction that overrides toString() to return asterisks.

The Vulnerable Flow:

1. Upstream Build (Build A): Defines db_password as type password. value: Hunter2!.
2. Downstream Build (Build B): references it via %dep.BuildA.db_password%.
3. Resolution: The agent resolves the dependency. Instead of treating the resolved value as a continuation of a secure object, the system effectively treated it as a standard string interpolation.

# Conceptually, the config looks like this:
object: BuildType
    id: DeployConfig
    dependency:
        id: BuildConfig
    parameters:
        # The fatal reference
        deploy_key = %dep.BuildConfig.secure_password%

When the agent spins up the build runner for DeployConfig, it prints the environment variables and parameters for debugging purposes. Because the 'password' attribute was attached to the definition in BuildConfig, the context in DeployConfig treated it as just another string coming from a dependency. The result in the log file:

[Step 1/1] Starting: /opt/deploy_script.sh
[Step 1/1] in directory: /opt/buildagent/work/...
[Step 1/1] Parameter 'deploy_key' = 'Hunter2!'  <-- OOPS.

The Fix (2023.05): The patch involves ensuring that the password property type is transitive. When a dependency parameter is resolved, the system now checks the source definition's type. If the source says password, the destination logger is instructed to add that value to the blacklist of strings to be masked, regardless of where it came from.

Exploiting this requires no hacking tools, no Metasploit, and no buffer overflows. It requires a web browser and a pair of eyes. The attack vector is strictly "Insider Threat" or "Compromised User."

The Scenario: Imagine you are a contractor or a junior developer. You have Project Viewer rights. You can't see the 'Administration' tab where the secrets are defined, and you can't edit the build configurations to echo secrets. However, you can view build logs to debug why a build failed.

1. Reconnaissance: Browse the project list for high-value targets. Look for "Deploy to Prod" or "Database Migration" build configurations.
2. Identify Dependencies: Look at the "Dependencies" tab. Note the upstream builds that likely hold the credentials (e.g., KeyVault_Fetch).
3. The Dig: Open a successful build of the downstream configuration. Click "Build Log".
4. The Loot: deeply nested in the verbose output, often in the "Resolving parameters" or "Agent properties" section, you will find the artifact dependency resolution logic.

[10:42:15] : [Resolving artifact dependencies]
[10:42:15] : ... resolved %dep.AuthService.api_token% to "ey...[FULL JWT OR API KEY]..."

Since the masking failed, the token is right there. You copy it, paste it into your local terminal, and now you have production access. It is the digital equivalent of finding the CEO's password written on a sticky note under their keyboard, except the keyboard is published to the company intranet.

The CVSS score of 4.3 is deceptively low. It assumes that if you are already inside the network (AV:N) and have an account (PR:L), the damage is limited (C:L). I fundamentally disagree with this assessment in a real-world context.

In modern DevOps, the CI/CD server is the Skeleton Key. It has write access to your git repositories, push access to your Docker registries, and deployment access to your Kubernetes clusters and AWS accounts. A single leaked credential here—say, a AWS_SECRET_ACCESS_KEY or a GITHUB_TOKEN with write permissions—allows for immediate lateral movement.

Real-World Consequences:

• Supply Chain Attack: An attacker uses the leaked git credentials to push malicious code into the repo, which the next build automatically deploys to production.
• Infrastructure Hijacking: Leaked Terraform or CloudFormation parameters allow the attacker to spin up crypto miners or delete critical infrastructure.
• Data Exfiltration: Database connection strings leaked in logs allow direct access to customer PII.

While the vulnerability requires authentication, in large organizations, "read access to logs" is often granted to hundreds of developers. Any one of them—or anyone who compromises one of their accounts—can harvest these credentials.

If you are running any version of TeamCity prior to 2023.05, you are vulnerable. The primary fix is simple: Upgrade. JetBrains has patched the resolution logic to ensure dependency parameters carry their 'password' status with them.

Immediate Remediation Steps:

1. Patch: Upgrade to TeamCity 2023.05 or later immediately.
2. Rotate: This is the painful part. You must assume that every secret passed as a dependency in your build chains has been compromised. Rotate your API keys, change your service account passwords, and revoke your SSH keys. If you don't do this, patching the server is useless—the thieves already have the keys.
3. Purge: Your historical logs are now toxic waste. Even after patching, the old logs on the disk still contain the plain text passwords. You need to write a script to scrub these logs or delete build history older than your patch date.

> [!WARNING] > Do not forget the Agent Logs. The leak happens on the build agent as well as the server. Ensure you purge teamcity-agent.log on all your build nodes, not just the central server UI logs.