Whatschat

How to Build and Scale a Hardened Container Image Pipeline: Lessons from Our First Year

Published: 2026-05-02 00:00:27 | Category: Cloud Computing

Introduction

One year ago, we launched Docker Hardened Images (DHI) with a mission to raise the security baseline of the entire container ecosystem. Since then, we’ve crossed 500,000 daily pulls, patched over 25,000 OS-level artifacts, and built a catalog of 2,000+ hardened images, MCP servers, Helm charts, and ELS images — all continuously updated. But the real story isn’t the numbers; it’s the deliberate, difficult choices we made to get here. This guide walks through the key steps we followed, so you can apply the same principles to your own hardened image pipeline. Whether you’re a security engineer, platform team lead, or DevOps practitioner, these steps will help you build a robust, trustworthy, and scalable approach to container security.

How to Build and Scale a Hardened Container Image Pipeline: Lessons from Our First Year
Source: www.docker.com

What You Need

  • A container registry (e.g., Docker Hub, Harbor, or a private registry) to store and distribute images.
  • A CI/CD pipeline that supports automated builds, testing, and signing (e.g., GitHub Actions, GitLab CI, Jenkins).
  • Source code for system packages — you’ll need to build from upstream source, not just repackage binaries, for true provenance.
  • Signing keys and a key management strategy to generate and store signing certificates for attestations (e.g., Sigstore, cosign).
  • SLSA Build Level 3 infrastructure — this requires trusted build platforms that generate provenance attestations (e.g., with slsa-github-generator or custom solutions).
  • A vulnerability scanning tool (e.g., Trivy, Grype) to identify CVEs and track patching progress.
  • An update mechanism — automated scripts or workflows to trigger rebuilds when new versions or patches are released.
  • Open source licensing — choose a permissive license (e.g., Apache 2.0) to maximize community adoption.
  • Documentation and release notes to communicate changes to your users.

Step-by-Step Guide

Step 1: Choose an Open Source Foundation

We made hardened images free and open source from day one. Security shouldn’t be a premium feature. By releasing under Apache 2.0, we enabled any team to use our work without negotiation. This step is critical because it builds trust and allows the community to inspect, verify, and contribute. Do not hide your catalog behind a paywall; instead, focus on building a sustainable model around support or premium tiers while keeping the core publicly available.

Step 2: Support Multiple Distributions

Some vendors create a proprietary “distroless” OS and force you to migrate. We took the opposite approach: we support Debian, Alpine, and other distributions you already use. This makes adoption drop-in — no migration tax, no retraining. For your pipeline, start with two to three popular distros and expand based on user demand. Ensure that each image is as close to the upstream distribution as possible, with minimal customizations, to maintain compatibility.

Step 3: Build All System Packages from Source

Don’t just copy binary packages from upstream repositories. Building from source gives you full control over compilation flags, dependencies, and patches. It also enables SLSA Build Level 3 attestations because the build process is fully deterministic and verifiable. For each distribution, maintain a source repository with patches and build scripts. Automate rebuilds when source versions change — we run over a million builds regularly to keep everything current.

Step 4: Implement SLSA Build Level 3

SLSA (Supply-chain Levels for Software Artifacts) Build Level 3 requires that your build system is fully hermetic, that you generate provenance attestations stating how the artifact was built, and that you cannot tamper with the build process. We achieved this by using trusted CI runners, recording every input and output, and signing provenance with a short-lived key. Follow the SLSA specification and use tools like slsa-github-generator to create verifiable attestations. This makes your images independently verifiable by anyone.

Step 5: Provide Comprehensive Attestations

Attestations are as important as the image itself. We ship SBOMs (Software Bill of Materials), provenance statements, vulnerability reports, and signatures with every image. Each attestation is signed and chained to the build process. When a user pulls our image, they can verify everything: where it came from, how it was built, what’s inside it, and what vulnerabilities are known. You should generate at minimum: an SBOM (CycloneDX or SPDX), a provenance attestation (in-toto or SLSA), and a cryptographic signature (e.g., cosign). Make these accessible via OCI references so they travel with the image.

How to Build and Scale a Hardened Container Image Pipeline: Lessons from Our First Year
Source: www.docker.com

Step 6: Continuously Patch and Rebuild

Static images are a security liability. We continuously monitor CVEs and upstream releases across all distributions and versions. Whenever a fix is available, we rebuild the affected images and push new attestations. This means we’re running over a million builds regularly. Automate this process: set up webhooks to trigger builds when distribution package repositories update, or use scheduled scans and rebuilds. Track your patching timeline — we aim to patch critical CVEs within 24 hours. Publish a public advisory and update the attestations accordingly.

Step 7: Grow Your Catalog Iteratively

Start with the most commonly used base images (e.g., Ubuntu, Debian, Alpine) and then expand to language runtimes, databases, and tools. We now have over 2,000 artifacts including Helm charts and MCP servers. Let community demand guide your roadmap. Maintain each artifact continuously — not just initial hardening. And always keep the foundation open so that adoption can scale organically.

Tips for Success

  • Evaluate patching timelines — when comparing hardened image providers, ask how fast they patch critical CVEs. Some take weeks; aim for days or hours.
  • Check SBOM completeness — a good SBOM should list all packages, versions, and licenses. Incomplete SBOMs hide risk.
  • Verify advisory coverage — do they publish advisories for every CVE? Are they transparent about false positives? This builds trust.
  • Don’t lock yourself into a proprietary distro — always prefer providers that support the distributions your team already uses. The “distroless” marketing often masks vendor lock-in.
  • Automate everything — manual patching doesn’t scale. Invest in CI/CD and monitoring.
  • Engage with the community — open source is a two-way street. Contributions and feedback improve your pipeline and increase adoption.
  • Plan for scale — our pipeline runs over a million builds per month. Ensure your infrastructure can handle rebuilds for multiple distros and versions simultaneously.

By following these steps, you can build a hardened image pipeline that matches the security and transparency we achieved. It’s not the easy path — but as we’ve proven over the past year, it’s the one that creates real, lasting impact.