Platform Engineering

Tutorial: Keyless Sign and Verify Your Container Images With Cosign

Just like supply chains of physical goods, the security of software supply chains is extremely important. Malicious attacks on the software supply chain are an ever-present threat and can cause extreme damage, which is evident from the Colonial Pipeline security attack which took down the largest fuel pipeline in the US due to a compromised password. 

The software supply chain is anything that affects your software, including everything that goes into writing and distributing your software: code, package manager, dependencies, who wrote it, who reviewed it, distribution mechanism, how it runs, where it runs, licensing … pretty much everything that touches it at any point.

Organizations need to be securing their software, so it's imperative to secure the software supply chain at every possible opportunity.

What's signing an image?

Simply put, signing an image provides cryptographic evidence that the author is who they say they are; based on them having access to the trusted private key and the content has not been changed since. Auth0 provides a great explanation on asymmetric public key cryptography, and also cover signatures.

Cryptographic signing has been around long before container images were a thing. I've personally taken pride in signing my git commits for over a decade (see how I manage my keys), but generating and protecting keys is not for the faint of heart.

Why sign images?

We live in a world with high-profile attacks on supply chains and an increasing requirement to demonstrate an auditable record of the provenance of where our assets have come from.

By simply authenticating to the registry to push while a strong first defense introduces further requirements for protecting those credentials and particularly when your CI system is creating the images those credentials may be visible to multiple team members.

This is then exasperated if you’re running your own registry since the administrators of that are likely administrators of other things, which lowers the barrier for how much collusion is required (too much privilege - trust is extended a long way).

Let's play a game of 'spot the difference': One of these images has something malicious in it that’ll do bad things when it runs (not really, it’ll just print “I am the bad guy” and then exit, but you get the point). The table shows the output of the Continuous Integration pipeline in GitHub and the corresponding images it built and pushed, storing it in Github Container Registry.

Build #7

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1

Unsiged Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417835412.1

Build #8

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417836769.1

Build #9

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417837856.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417837856.1

Build #10

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417838926.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417838926.1

Spot anything by just looking at the unsigned images? 

By looking at the metadata of the build outputs it is hard to tell if any of these images have been changed since they were built from the CI pipeline. Spoiler alert: It's Build #7.

Okay, maybe I made this one easy since the time it took me to write the bad code, push and sign it under my own signature makes it stand out in the registry.

One way of checking if the image has been tampered with is to use a utility called cosign. If we take the signed image name from Build #8 and run the following command:

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1 | jq

The following output is shown:

"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-demo-spotthedifference/.github/workflows/ci.yml@refs/heads/main"

If we run the same command but this time with the image from Build #7, you can see that the Issuer and subject are different.

$COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1 | jq
     "Issuer": "https://github.com/login/oauth",
     "Subject": "chris@cns.me.uk"

You can verify this yourself by installing cosign and running against any of the images in our demonstration repo. In the two outputs above, it can be seen image in Build #8 was created by a pipeline and the image from Build #7 was created by a bad actor aka chris@cns.me.uk.

Let’s not forget that if a bad actor has write access to your registry (or the ability to impersonate it) then they don’t need to leave their bad image there for long. They can always replace it with the good one later after they’ve reached their goal of it running in your cluster; hence while signing is important, constantly verifying that is how you unlock the value.

Traditional issues with keypair signing

When creating and pushing signed images to a container registry, two things are required: registry credentials and a private key. You’ve now got two problems: Securing registry credentials and securing a private key. Inevitably you’ll keep these in a similar way, a compromised device such as a developer laptop or CI system quickly negates any of the benefits of signing in the first place.

It’s possible to use hardware tokens such as a Yubikey for humans which can be configured to require a PIN and physically touch the token to sign anything. It is not possible to extract the private key material from the device. However, this is inherently not practical to implement for your non-human fingerless builders.

Enter: Cosign

Cosign from sigstore makes the traditional signing of container images vastly easier. However, this can be taken a step further in combination with the wider sigstore products. Using fulcio and rekor, it's possible to sign images (and other things too) without keys and later verify them against an immutable transparency log.

So, how do you sign without keys?

Okay, we don’t really sign without keys. But behind the scenes, cosign creates the keypair ephemerally (they last 20 minutes) and gets them signed by Fulcio using your authenticated OIDC identity.

OIDC

OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorisation Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

OIDC with human interaction

https://www.youtube.com/watch?v=a0KC3DPLD2s

$ COSIGN_EXPERIMENTAL=1 cosign sign chrisns/cosign-keyless-demo:latest

$ COSIGN_EXPERIMENTAL=1 cosign verify chrisns/cosign-keyless-demo:latest

Note the issuer and subject, this is from the identity Fulcio received and stored in the Rekor transparency log.

"Issuer": "https://accounts.google.com",
"Subject": "chris.nesbitt-smith@appvia.io"

OIDC flow with interaction-free with GitHub actions

Now lets do it without interaction, using the following workflow:

name: Build Push Sign
on: { push: { branches: ['main'] } }

jobs:
 build:
   runs-on: ubuntu-latest
   permissions:
     packages: write
     id-token: write

   steps:
     - uses: actions/checkout@v2.3.5

     - name: Login to GitHub
       uses: docker/login-action@v1.9.0
       with:
         registry: ghcr.io
         username: ${{ github.actor }}
         password: ${{ secrets.GITHUB_TOKEN }}

     - name: build+push
       uses: docker/build-push-action@v2.7.0
       with:
         push: true
         tags: ghcr.io/chrisns/cosign-keyless-demo:latest

     - uses: sigstore/cosign-installer@main

     - name: Sign the images
       run: |
         cosign sign ghcr.io/chrisns/cosign-keyless-demo:latest
       env:
         COSIGN_EXPERIMENTAL: 1

Now your images will have something like the following when you run:

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-keyless-demo:latest

"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main"

What this allows us to do is cryptographically demonstrate that the image we can pull was built by the CI pipeline of that repository. Neat!

Who should sign their containers?

As we’ve seen, signing your container images can be as trivial as a couple of lines added to your CI pipeline.

If you’re a software vendor that ships their product as a container, signing is a great way to reassure your customers that you take security seriously and allow them to verify the authenticity of your product.

If you’re building software for your own business use, it’s also a great way to validate the integrity of your own supply chain internally.

Verifying the OIDC signed image

So now that we can verify the image manually, the next logical step is to let our Kubernetes cluster do that for us.

Kubernetes admission controller

In Kubernetes, an Admissions Controller is a piece of code that intercepts requests to the API server and does something with it. We have created an admissions controller that can either approve or deny the image from running in the cluster based on its cryptographic signature. 

For the next step, you'll need a Kubernetes cluster. If you don’t have one to hand, you can use KiND or minikube.

Installation

# if you don't already have cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -k https://github.com/appvia/cosign-keyless-admission-webhook

Usage

In your pod spec you set an annotation(s) of subject.cosign.sigstore.dev/CONTAINERNAME* to the subject of the certificate and also set the issuer.cosign.sigstore.dev/CONTAINERNAME* to the Issuer.

*CONTAINER_NAME is the name of the container from your pod specification.

Full example:

apiVersion: v1
kind: Pod
metadata:
 annotations:
   subject.cosign.sigstore.dev/demo: https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main
   issuer.cosign.sigstore.dev/demo: https://token.actions.githubusercontent.com
 name: cosign-keyless-demo
spec:
 containers:
   - image: ghcr.io/chrisns/cosign-keyless-demo:latest
     name: demo

Save the contents above in a file called pod.yaml and run: 

$ kubectl apply -f pod.yaml

Since the image ghcr.io/chrisns/cosign-keyless-demo:latest is signed as we expect the Admission controller to accept the request and the pod will be accepted.

And if you change the pod.yaml to:

apiVersion: v1
kind: Pod
metadata:
 annotations:
   subject.cosign.sigstore.dev/demo: chris@cns.me.uk
   issuer.cosign.sigstore.dev/demo: "https://github.com/login/oauth",
 name: cosign-keyless-demo-bad
spec:
 containers:
   - image: ghcr.io/chrisns/cosign-keyless-demo:latest
     name: demo

You'll find it will be rejected by the admission controller and not permitted to run on the cluster as this image is not signed as expected.

Known limitations and potential improvements

  • Untested with registries that require authentication
  • No caching, so it checks each pod, which can be slow
  • Requires connectivity to the transparency log and the registry
  • Wildcards and more complex rules are defined on the namespace or potentially cluster level
  • Doesn’t check signatures on initContainers

Start signing your images

If you’ve followed along with the above you can now sign your images, and check the signatures in Kubernetes before accepting the pod to run, assuring that the image your cluster is about to start hasn’t been tampered with. Ensuring that the images form part of a secure software supply chain from creation to runtime.

The admission controller I made is open-source on GitHub, so please do star and watch it, and as always pull requests are always very welcome.

Latest Articles