[Infrastructure as Code (IaC)] GitLab Runner, Kubernetes executor and kaniko to build container images from a Dockerfile inside a Kubernetes (K8S)

Posted on 2021-10-09 Edited on 2023-09-02 In Infrastructure as Code (IaC) , GitLab , GitLab Runner Views: Word count in article: 6.5k Reading time ≈ 6 mins.

Kaniko

GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.

kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

The Kubernetes executor, when used with GitLab CI, connects to the Kubernetes API in the cluster creating a Pod for each GitLab CI Job.

kaniko solves two problems with using the Docker-in-Docker https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-docker-executor-with-the-docker-image-docker-in-docker build method:

Docker-in-Docker requires privileged mode to function, which is a significant security concern.
Docker-in-Docker generally incurs a performance penalty and can be quite slow.
Requirements

To use kaniko with GitLab, a runner with one of the following executors is required:

Kubernetes The Kubernetes executor | GitLab - https://docs.gitlab.com/runner/executors/kubernetes.html.

Building a Docker image with kaniko

When building an image with kaniko and GitLab CI/CD, you should be aware of a few important details:

The kaniko debug image is recommended (gcr.io/kaniko-project/executor:debug) because it has a shell, and a shell is required for an image to be used with GitLab CI/CD.
The entrypoint needs to be overridden, otherwise the build script doesn’t run.
A Docker config.json file needs to be created with the authentication information for the desired container registry.

In the following example, kaniko is used to:

Build a Docker image.
Then push it to GitLab Container Registry.

The job runs only when a tag is pushed. A config.json file is created under /kaniko/.docker and /root/.docker/config.json with the needed GitLab Container Registry credentials taken from the predefined CI/CD variables GitLab CI/CD provides.

In the last step, kaniko uses the Dockerfile under the root directory of the project, builds the Docker image and pushes it to the project’s Container Registry while tagging it with the Git tag:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json

    - cp /kaniko/.docker/config.json /root/.docker/config.json
    
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
  rules:
    - if: $CI_COMMIT_TAG

Additional Flags

–build-arg

This flag allows you to pass in ARG values at build time, similarly to Docker. You can set it multiple times for multiple arguments.

–dockerfile

Path to the dockerfile to be built. (default “Dockerfile”)

–no-push

Set this flag if you only want to build the image, without pushing to a registry.

See - https://github.com/GoogleContainerTools/kaniko#additional-flags to learn more.

Advances

Using a registry with a custom certificate

When trying to push to a Docker registry that uses a certificate that is signed by a custom CA, you might get the following error:

1
2
3

$ /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --no-push
INFO[0000] Downloading base image registry.gitlab.example.com/group/docker-image
error building image: getting stage builder for stage 0: Get https://registry.gitlab.example.com/v2/: x509: certificate signed by unknown authority

This can be solved by adding your CA’s certificate to the kaniko certificate store:

before_script:
  - mkdir -p /kaniko/.docker
  - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
  - |
    echo "-----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----" >> /kaniko/ssl/certs/additional-ca-cert-bundle.crt

Runner Kubernetes executor DNS

Pod’s DNS Config gives users more control on the DNS settings of the created build Pods.

concurrent = 1
check_interval = 30
[[runners]]
  name = "myRunner"
  url = "https://gitlab.example.com"
  token = "__REDACTED__"
  executor = "kubernetes"
  [runners.kubernetes]
    image = "alpine:latest"
    [runners.kubernetes.dns_config]
      nameservers = [
        "1.2.3.4",
      ]
      searches = [
        "ns1.svc.cluster-domain.example",
        "my.dns.search.suffix",
      ]

      [[runners.kubernetes.dns_config.options]]
        name = "ndots"
        value = "2"

      [[runners.kubernetes.dns_config.options]]
        name = "edns0"

See The Kubernetes executor | GitLab - https://docs.gitlab.com/runner/executors/kubernetes.html#pods-dns-config to learn more.

Or replace /etc/resolv.conf in shell script.

1
2
3

before_script:
  - echo 'nameserver 8.8.8.8' > /etc/resolv.conf && echo 'nameserver 1.1.1.1' >> /etc/resolv.conf
#   - cat /etc/resolv.conf

Use statically-defined credentials

There are two approaches that you can take to access a private registry. Both require setting the CI/CD variable DOCKER_AUTH_CONFIG with appropriate authentication information.

Per-job: To configure one job to access a private registry, add DOCKER_AUTH_CONFIG as a CI/CD variable.
Per-runner: To configure a runner so all its jobs can access a private registry, add DOCKER_AUTH_CONFIG as an environment variable in the runner’s configuration.

See Use statically-defined credentials | Run your CI/CD jobs in Docker containers | GitLab - https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#use-statically-defined-credentials

FAQs

Variable value is empty

If the variable is marked as protected, please remember to also mark the corresponding branch as protected, otherwise the obtained variable value is empty.