Development Inside Kubernetes Part 5: Kaniko

In Development Inside Kubernetes Part 4: Private Container Registry, I was able to setup a private container registry which is actually important to a local development environment. Typically, people just use a local images cache inside of docker, but the best solution is a private registry. It leads to better outcomes because you are always working with a registry which is exactly what you will do in a live environment.

The next step is to develop or be able to build containers from within the cluster itself. There are 3 basic use cases we want to attempt.

Setup

Initially, I need to give kubernetes pods access in some way to files like container descriptors. This means any Dockerfiles or paths to Dockerfiles will be accessible from my kaniko pod. Here, I setup a PersistentVolume and point it to a path in my home directory where I intend to give access to Dockerfiles.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: dockerfile
  labels:
    type: local
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: local-storage
  hostPath:
    path: /Users/<user-name>/kaniko # replace with local directory, such as "/home/<user-name>/kaniko"

A claim needs to be created to bind it. For both, the PersistentVolume and PersistentVolumeClaim a local-storage storageClassName is used.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dockerfile-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage

Besides PersistantVolume and PersistantVolumeClaim, we will need a secret. This is already covered in [Development Inside Kubernetes Part 4: Private Container Registry])@timesheets/5kztcs-development-inside-kubernetes-part-4-private-container-registry). I am going to reuse my docker secret I created there.

Usage

One of the great things about kaniko is that nothing needs to be installed. It can run in a Pod by itself. I am using the container gcr.io/kaniko-project/executor:debug.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh","-c","cat $STACKLOG_PATH"]
    env:
      - name: STACKLOG_PATH
        value: /workspace/kaniko.slog
    args: ["--dockerfile=/workspace/dockerfile",
            "--context=dir://workspace",
            "--destination=registry.local.thelastpri.me/test/cloudtools"] # replace with your dockerhub account
    volumeMounts:
      - name: kaniko-secret
        mountPath: /kaniko/.docker
      - name: dockerfile-storage
        mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: dockerfile-claim

For more information, you can checkout the kaniko getting started tutorial. At this point, I have not really said anything more than what is already in the tutorial.

How I am Using Kaniko

What I am really trying to do is develop in such a way that replaces my docker command. Kaniko supports stdin input, but I decided against using this because of context. I like to build from context. I have assets and dependencies that I need to build with. It is possible to gzip context and pass in through stdin, but I would rather instead.

apiVersion: v1
kind: Job
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh","-c","cat $STACKLOG_PATH"]
    env:
      - name: STACKLOG_PATH
        value: /workspace/kaniko.slog
    args: ["--dockerfile=/workspace/dockerfile",
            "--context=dir://workspace",
            "--destination=registry.local.thelastpri.me/test/cloudtools"] # replace with your dockerhub account
    volumeMounts:
      - name: kaniko-secret
        mountPath: /kaniko/.docker
      - name: dockerfile-storage
        mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: dockerfile-claim

Instead of a Pod, I use a Job.

Developing on WSL

Using rancher desktop on WSL will create a VM in addition to your wsl environment called rancher-desktop.

PS C:\Users\timesheets> wsl -l
Windows Subsystem for Linux Distributions:
Debian (Default)
rancher-desktop-data
rancher-desktop

All wsl environments have access to /mnt/wsl/rancher-desktop

❯ ls -l /mnt/wsl/rancher-desktop/run/data
total 64
drwxr-xr-x 2 root root  4096 Sep 20 11:24 bin/
drwxr-xr-x 2 root root  4096 Sep 20 11:24 dev/
drwxr-xr-x 3 root root  4096 Nov  5 19:56 etc/
-rwxr-xr-x 1 root root     0 Nov  5 19:56 init*
drwxr-xr-x 4 root root  4096 Aug  2 15:13 lib/
drwx------ 2 root root 16384 Apr 10  2019 lost+found/
drwxr-xr-x 6 root root  4096 Sep 27 11:40 mnt/
drwxr-xr-x 2 root root  4096 Sep 20 11:24 proc/
drwxr-xr-x 2 root root  4096 Sep 20 11:24 run/
drwxr-xr-x 2 root root  4096 Sep 20 11:24 sbin/
drwxr-xr-x 2 root root  4096 Sep 20 11:24 sys/
drwxrwxrwt 3 root root  4096 Oct 12 05:01 tmp/
drwxr-xr-x 4 root root  4096 Oct  3 21:15 usr/
drwxr-xr-x 3 root root  4096 Sep 20 11:24 var/

This commonly shared volume allows me to sync my workspace with a volume that kaniko will have access to. I just set the shared volume as my PersistentVolume.

Then, I can create my builds by using

rsync -av workspace/* /mnt/wsl/rancher-desktop/usr/src/workspace
kubectl apply -f kaniko.yaml

In the end, kaniko was adequate, but I needed something more consistent. Kaniko is still being developed and things like multistage builds are not well-supported.