.putty P1DocsCloud Computing
Related
How to Design Scalable Cloud Storage Like Amazon S3: A 20-Year Legacy GuideDynamic Workflows: Durable Execution Customized Per Tenant10 Essential Insights into Cloudflare's Dynamic Workflows: The Future of Multi-Tenant Durable ExecutionTailoring Cloud Service Dashboards in Grafana Cloud: Customize AWS, Azure, and GCP ViewsWhat You Need to Know About AWS Weekly Roundup: Claude Opus 4.7 in Amazon Bed...Analyzing the AI-Native Spending Boom: A Strategic Guide for Enterprise Software Leaders10 Essential Principles for Cloud Cost Optimization in the Age of AIInside the Pentagon's $17.9 Billion Golden Dome Laser Defense Program

The Silent Upgrade: How Kubernetes Image Promotion Got a Modern Makeover

Last updated: 2026-05-01 14:36:04 · Cloud Computing

Every container image you pull from registry.k8s.io arrives thanks to kpromo, the Kubernetes image promoter. This tool copies images from staging registries to production, signs them with cosign, replicates signatures across over 20 regional mirrors, and generates SLSA provenance attestations. If kpromo breaks, no Kubernetes release ships. Recently, the team rewrote its core from scratch, deleted 20% of the codebase, made it dramatically faster, and nobody noticed. That was the goal.

A Brief History

The image promoter began in late 2018 as an internal Google project by Linus Arver. The aim was to replace manual, Googler-gated image copying into k8s.gcr.io with a community-owned GitOps workflow. Push to a staging registry, open a PR with a YAML manifest, get reviewed and merged, and automation handles the rest. KEP-1734 formalized this.

The Silent Upgrade: How Kubernetes Image Promotion Got a Modern Makeover

In early 2019, the code moved to kubernetes-sigs/k8s-container-image-promoter and grew quickly. Stephen Augustus consolidated multiple tools (cip, gh2gcs, krel promote-images, promobot-files) into a single CLI called kpromo. The repo was renamed to promo-tools. Adolfo García Veytia (Puerco) added cosign signing and SBOM support. Tyler Ferrara built vulnerability scanning. Carlos Panato maintained the project. 42 contributors made ~3,500 commits across 60+ releases. It worked, but by 2025 the codebase carried seven years of incremental additions. The README noted duplication, multiple techniques, and many TODOs.

Problems to Solve

Production promotion jobs for Kubernetes core images regularly took over 30 minutes and often failed with rate limit errors. The core promotion logic became a monolith hard to extend and test, making new features painful. On the SIG Release roadmap, two work items lingered: "Rewrite artifact promoter" and "Make artifact validation more robust." Discussions at meetings and KubeCons led to open research spikes on project board #171 capturing eight questions.

The Phased Rewrite

In February 2026, the team opened issue #1701 to rewrite the artifact promoter pipeline. All eight spikes were answered in a single tracking issue. The rewrite was phased so each step could be reviewed, merged, and validated independently.

Phase 1: Rate Limiting (#1702)

Rewrote rate limiting to properly throttle all registry operations with adaptive backoff.

Phase 2: Interfaces (#1704)

Put registry and auth operations behind clean interfaces for independent swapping and testing.

Phase 3: Pipeline Restructuring

The pipeline was restructured into composable stages, allowing easier extension for provenance and vulnerability scanning. This modular approach reduced coupling and simplified testing.

Conclusion: Invisible Improvements

The rewrite made kpromo faster, more reliable, and easier to maintain. The 20% code deletion and new architecture ensure future features can be added without breaking existing workflows. The team succeeded in making the improvements invisible to users — exactly as intended.