Why Use a Registry Mirror?

As of April 1, 2025, Docker has implemented new pull rate limits for Docker Hub. These changes may significantly impact Kubernetes Clusters.

Here’s a quick overview of the updated limits:

User Type Pull Rate Limit (per 6 hours) Public Repositories Private Repositories
Business (authenticated) Unlimited Unlimited Unlimited
Team (authenticated) Unlimited Unlimited Unlimited
Pro (authenticated) Unlimited Unlimited Unlimited
Personal (authenticated) 200 Unlimited Up to 1
Unauthenticated users 100 per IP N/A N/A

When unauthenticated, Docker Hub will limit you to 100 pulls every 6 hours per IPv4 address or IPv6 /64 subnet.

For more details, check the official Docker Hub pull usage and limits documentation.

To avoid these limitations and make your Kubernetes clusters more resilient and performant, setting up a local registry mirror is highly recommended.

One great option: the k3s embedded registry mirror.

Setting Up the k3s Embedded Registry Mirror

As of December 2024, the embedded registry mirror in k3s is GA (Generally Available) in the following versions:

  • v1.29.12+k3s1
  • v1.30.8+k3s1
  • v1.31.4+k3s1

Earlier versions from January 2024 had it available as an experimental feature.

The embedded mirror is powered by Spegel, a stateless distributed OCI registry mirror that allows peer-to-peer sharing of container images across your k3s cluster.

1. Enable the Distributed OCI Registry Mirror

You must start the server nodes with the embedded registry enabled. Add the following to your k3s config file or CLI options:

# /etc/rancher/k3s/config.yaml
embedded-registry: true

Alternatively, using the CLI:

k3s server --embedded-registry

Once enabled:

  • Every node hosts a local OCI registry on port 6443.
  • Nodes advertise available images over a peer-to-peer network on port 5001.

You can change the peer-to-peer port by setting the K3S_P2P_PORT environment variable, but this is not recommended.

Important: All nodes must be able to reach each other over their internal IPs on TCP ports 5001 and 6443.

2. Enable Registry Mirroring

To mirror images from Docker Hub (docker.io) or Kubernetes Registry (registry.k8s.io), edit or create the registries.yaml file on all nodes:

# /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
  registry.k8s.io:

If you prefer to mirror all registries automatically, wildcard support is available:

mirrors:
  "*":

Note: You must quote the * wildcard!

3. Special Handling for latest Tag

Images pulled with the latest tag are not distributed between nodes by default.

To mirror images tagged as latest, set this environment variable:

K3S_P2P_ENABLE_LATEST=true

Warning: Enabling this is unsupported and not recommended because latest tags can frequently change.

4. Security Considerations

  • Access to the embedded registry requires a client certificate signed by the cluster’s CA.
  • The peer-to-peer network uses a pre-shared key and mutual certificate authentication.
  • Be aware: All nodes can share images, even if some images were originally private.
  • To avoid potential tampering, always reference images by digest, not by mutable tags.

5. Sharing Air-gap or Manually Loaded Images

You can manually load images into the embedded registry using ctr:

ctr -n k8s.io image import /path/to/image.tar

or pull directly:

ctr -n k8s.io image pull docker.io/library/nginx:latest

Tip: Always use the k8s.io namespace when managing images with ctr so they are visible to Kubernetes.

6. Monitoring Spegel Metrics

Spegel, the engine behind the embedded registry mirror, exposes Prometheus-compatible metrics. These help you monitor the health and performance of your image distribution.

Here are the key metrics you should care about:

Metric Type Why It Matters
spegel_advertised_images Gauge Shows how many images your node is sharing with others. A sudden drop might indicate problems with local images or networking.
spegel_mirror_requests_total Counter Counts image pull requests handled by Spegel, with labels like cache=hit or missandsource=internal external. Use it to measure how effective the mirror is. High miss counts may suggest missing images.
spegel_resolve_duration_seconds Histogram Tracks how long it takes to resolve image requests. Spikes here might indicate network issues or slow storage.
http_request_duration_seconds Histogram Measures how long HTTP requests to the embedded registry take. Useful for spotting performance bottlenecks.
http_requests_inflight Gauge Shows how many requests are currently being processed. High values could hint at overloaded nodes.

Example Usage

  • Detect issues early: If spegel_advertised_images suddenly drops to zero, the node might have lost its local images or network connectivity.
  • Tune performance: Monitor spegel_resolve_duration_seconds and http_request_duration_seconds to ensure fast image pulls.
  • Validate caching efficiency: Look at the cache=hit vs cache=miss labels in spegel_mirror_requests_total. A high hit ratio means your cluster is serving images efficiently.

You can scrape these metrics directly by pointing Prometheus to the k3s embedded registry metrics endpoint (usually exposed on the node’s IP and registry port).

Tip: Consider setting alerts on low cache=hit rates or high resolve_duration_seconds to catch issues automatically.

Conclusion

Setting up the k3s embedded registry mirror is an excellent way to:

  • Avoid Docker Hub pull rate limits
  • Speed up container pulls inside the cluster
  • Make your clusters more resilient in case of external outages

With just a few configuration changes, you can fully leverage the power of peer-to-peer image distribution in k3s.

Happy clustering! 🚀