Skip to main content
ASM Cheatsheet
๐Ÿ”ดAdvanced
Incident Replay
Real incident
18 min

A Credential in a Docker Image

The 2021 Codecov Bash Uploader supply-chain compromise

This one runs backwards from most breaches: the victim organization was never the target. An attacker extracted a credential that had been baked into a public Docker image, used it to modify a script that thousands of CI pipelines download and execute on every build, and let those pipelines exfiltrate their own environment variables โ€” AWS keys, deploy keys, API tokens, service-account credentials. The modification began 31 January 2021 and ran undetected until 1 April, when a customer compared the script's checksum against the one published on GitHub and found they did not match. Every downstream organization was compromised by trusting a build tool. Press play, then switch to Defend mode.

Source: Codecov โ€” Post-Mortem / Root Cause Analysis (April 2021)

Educational reconstruction based on Codecov's published post-mortem and root cause analysis. This replay is event-counted rather than action-counted: the public record supports a sequence of events, not a per-step action tally. Dates come from the vendor's disclosure; the commands shown are representative illustrations of the documented techniques, not recovered artifacts.

Break the Chain

You're the defender. Deploy up to 2 controls, then run the intrusion. Each control you place severs the attack chain at one point โ€” the agent reaches everything up to it, and nothing beyond. Contain it as early as you can.

Defensive controls0/2 deployed
The attack chain

Deploy controls, then run to see how far the agent gets.

  1. Public registry
    Public Docker image
    vendor-published ยท contains an HMAC key
  2. Vendor cloud (GCP)
    โ†“extract key
    Service-account credential
    write access to the artifact bucket
  3. โ†“write access
    Artifact storage
    hosts the Bash Uploader script
  4. Vendor distribution
    โ†“modify script
    Bash Uploader script
    curl | bash on every build
  5. Downstream customers ยท CI
    โ†“downloaded per build
    Customer CI pipelines
    thousands of builds executing the script
  6. โ†“read environment
    CI environment
    AWS keys ยท deploy keys ยท API tokens
  7. Attacker infrastructure
    โ†“exfiltrate
    Collection server
    receives exported environment

Defensive lessons

  • 1Never bake credentials into container images. Image layers are immutable and public images are inspectable โ€” a secret present during any build step is recoverable forever, regardless of whether a later step deletes it. Scan images for secrets in CI, the same way you scan source.
  • 2`curl | bash` is a standing grant of code execution to whoever controls that URL. If a build must fetch a script, pin it by digest and verify a signature or checksum before executing โ€” the vendor published a SHA256 precisely so this could be caught, and one customer checking it is what ended the incident.
  • 3The credential that publishes your artifacts is a production secret. It deserves separation, rotation, and monitoring at the same level as your customer database, because it can reach every one of your customers at once.
  • 4CI environment variables are a credential store with no access control. Any code the pipeline runs can read all of them. Prefer short-lived, workload-scoped credentials (OIDC federation) over long-lived static keys, so a stolen environment expires quickly.
  • 5Your attack surface includes your suppliers' attack surface. None of the downstream victims were compromised through their own perimeter โ€” they were compromised by a build tool they trusted. Inventory what your pipelines execute and where it comes from.
  • 6Integrity monitoring beats perimeter monitoring for supply-chain attacks. Nothing looked anomalous: a build fetched its normal tool from its normal URL. Only comparing the artifact against its expected hash exposed it.
  • 7Assume detection will be slow and plan the recovery. Two months of collection meant the response was mass credential rotation across many organizations. Knowing which secrets are exposed to CI, and being able to rotate them quickly, is the control that limits the damage.