Development Inside Kubernetes Part 4: Private Container Registry

In Development Inside Kubernetes Part 3: Let's Encrypt Dashboard that was just the warmup. I am now going to use the exact same process to setup and deploy my container registry. I will need a private/local container registry to use for my local development environment.

Recall that I opted for containerd rather than moby/docker to manage my local container images. This means, I do not have builtin docker features. I have to push my container images somewhere when I build them. I am opting for a local private registry to do this. Docker has their own registry, but I may switch to another. For now, docker's registry is the easiest to setup. The path to least resistance.

Docker Secret

For HTTP auth, I need to create a set of credentials, then add it to a secret for kubernetes. To create a password, I can generate one from /dev/urandom and use the docker registry container to create the htpasswd file.

export PASS=$(head -c 16 /dev/urandom | shasum | cut -d " " -f 1)  # Generates random pw
export USER=admin

echo $USER > creds.txt
echo $PASS >> creds.txt

docker run --entrypoint htpasswd registry:2 -Bbn admin $PASS > ./htpasswd  # Uses an older container image because newer images don't have htpasswd

unset $PASS $USER

There is a helm chart for docker-registry that allows pretty much any case of private container registry installation. It even includes ingress which I am going to replace because I do not care for it.

helm repo add stable https://charts.helm.sh/stable          # Helm 3 requires we add the repo for the stable repository
helm install stable/docker-registry \
  --name registry \
  --namespace default \                                     # kube-system is also a good place for this
  --set persistence.enabled=true \                          # This enables a PVC
  --set secrets.htpasswd=$(cat ./htpasswd)

Traefik for Ingress

First, we need to setup ingress. It points to a secret with the same name as the registry deployment. In my case, I called it registry

❯ kubectl get secrets
NAME                                       TYPE                 DATA   AGE
registry-internal-docker-registry-secret   Opaque               2      21m
sh.helm.release.v1.registry-internal.v1    helm.sh/release.v1   1      21m

I can see the secret name created from the helm deployment. I can pass that off to Traefik as a middleware for auth.

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: registry-auth
  namespace: default
spec:
  basicAuth:
    secret: registry-internal-docker-registry-secret

The next piece requires me to setup the Traefik routes. Notice that for SSL, I am reusing my certificate from before local-acme-crt

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host("registry.local.thelastpri.me")
      services:
        - kind: Service
          port: 5000
          name: registry-internal-docker-registry
          namespace: default
      middlewares:
        - name: registry-auth
    - kind: Rule
      match: Host("registry.local.thelastpri.me") && PathPrefix("/v2")
      services:
        - kind: Service
          port: 5000
          name: registry-internal-docker-registry
          namespace: default
      middlewares:
        - name: registry-auth
  tls:
    secretName: local-acme-crt

Next time, I am going to setup buildkit and kaniko to push out a build.