Summary
This post shows you how to retrieve the list of container images from a live Kubernetes cluster, and check the manifests of those images at their registries to see whether an arm64/aarch64 version is available.
Background
The arm64/aarch64 platform is gaining popularity in the container world as the chips themselves spread to more domains. They've been used on single board computers like the Raspberry Pi and smartphones for a while, but more recently on Apple Macs using Apple Silicon and various ARM-based cloud offerings from AWS, Google Cloud, Azure and Hetzner (who also offer dedicated servers using Ampere Altra chips). The offerings are generally cheaper than Intel/AMD equivalents and AWS and Google Cloud are even offering their basic ARM VMs on a free trial basis (other charges may apply, like storage and bandwidth).
Can I move my Kubernetes workloads to ARM?
With the substantial potential cost savings, it's worth considering whether to move Kubernetes workloads onto arm64 instances, either fully or as hybrid clusters. As a first step, you'll want to check whether your container images are available for arm64 in the first place.
Here, we'll create a one-liner shell script to scan through container images running on a Kubernetes cluster, checking each one with its registry to see if an arm64 version is available. I'll go through how I assembled the one-liner to help show how it works, but of course you can skip to the end if you just want to copy/paste.
Step 1: Retrieve a list of running container images on Kubernetes
The first step is to retrieve the list of all container images running on Kubernetes. The Kubernetes docs have a post on this, which gives this one-liner:
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
Step 2: How to check a Docker/container image for arm64 compatibility
From the Docker docs, you can check a Docker/container image for arm64 compatibility using Docker buildx like so:
docker buildx imagetools inspect --format '{{json (index .Image "linux/arm64")}}' moby/buildkit:master
Step 3: Piping the list of containers into docker buildx
Using these two snippets, we can pipe the list of containers into docker buildx to check them for arm64 compatibility.
But first, you may notice that some containers outputted by step 1 have a sha256 hash on the end, which is platform specific, e.g.:
quay.io/cilium/cilium:v1.13.1@sha256:428a09552707cc90228b7ff48c6e7a33dc0a97fe1dd93311ca672834be25beda
This is easily fixed with a quick sed to find/replace:
sed "s/@sha.*//"
We can then pipe the result into a while loop to check each image (I checked xargs as an option for the loop, but the slightly different implementations across BSD (Mac) and GNU (Linux) make it difficult to do it portably). If we put the loop in parentheses, it runs as a single process, making it easier to Ctrl+C to interrupt. Finally, I added some shell escapes to add colours and bolding, to make the output easier to read.
The final command
Here's the final command to check all Kubernetes container images for arm64 compatibility:
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq |\
sed "s/@sha.*//" |\
(while read image; do docker buildx imagetools inspect --format '{{json (index .Image "linux/arm64")}}' $image >/dev/null 2>&1 && echo -e "$image \033[1;92mOK\033[0m" || echo -e "$image \033[1;91mNOT-AVAILABLE\033[0m"; done)
And here is some example output, trimmed for clarity (in a terminal, OK shows in green and NOT-AVAILABLE shows in red):
docker.io/coredns/coredns:1.10.0 OK
quay.io/cilium/cilium:v1.13.1 OK
quay.io/cilium/operator-generic:v1.13.1 OK
quay.io/prometheus-operator/prometheus-config-reloader:v0.63.0 OK
quay.io/prometheus-operator/prometheus-operator:v0.63.0 OK
quay.io/prometheus/alertmanager:v0.25.0 OK
quay.io/prometheus/node-exporter:v1.5.0 OK
quay.io/prometheus/prometheus:v2.42.0 OK
registry.k8s.io/kube-apiserver:v1.26.3 OK
registry.k8s.io/kube-controller-manager:v1.26.3 OK
registry.k8s.io/kube-proxy:v1.26.3 OK
registry.k8s.io/kube-scheduler:v1.26.3 OK
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.2 OK
registry.k8s.io/metrics-server/metrics-server:v0.6.3 OK
registry.k8s.io/sig-storage/csi-attacher:v4.1.0 OK
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0 OK
registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 OK
registry.k8s.io/sig-storage/csi-resizer:v1.7.0 OK
registry.k8s.io/sig-storage/livenessprobe:v2.9.0 OK
restic/restic:0.15.1 NOT-AVAILABLE
In this case, only the restic/restic image is not available for arm64.
Conclusion
I've shown how to check a list of Kubernetes container images from a live cluster for arm64 compatibility. The approach could be extended to other architectures, or the list of images could be read from elsewhere (e.g. a file) with some small adaptations. This helps with planning before adopting arm64 nodes into a cluster, or indeed porting to arm64 completely where feasible.
I hope you found this post useful. Thanks for reading!
Bluesky