Tutorial: Keyless Sign and Verify Your Container Images With Cosign

Table of Contents

Just like supply chains of physical goods, the security of software supply chains is extremely important. Malicious attacks on the software supply chain are an ever-present threat and can cause extreme damage, which is evident from the Colonial Pipeline security attack which took down the largest fuel pipeline in the US due to a compromised password. 

The software supply chain is anything that affects your software, including everything that goes into writing and distributing your software: code, package manager, dependencies, who wrote it, who reviewed it, distribution mechanism, how it runs, where it runs, licensing … pretty much everything that touches it at any point.

Organizations need to be securing their software, so it’s imperative to secure the software supply chain at every possible opportunity.

What’s signing an image?

Simply put, signing an image provides cryptographic evidence that the author is who they say they are; based on them having access to the trusted private key and the content has not been changed since. Auth0 provides a great explanation on asymmetric public key cryptography, and also cover signatures.

Cryptographic signing has been around long before container images were a thing. I’ve personally taken pride in signing my git commits for over a decade (see how I manage my keys), but generating and protecting keys is not for the faint of heart.

Why sign images?

We live in a world with high-profile attacks on supply chains and an increasing requirement to demonstrate an auditable record of the provenance of where our assets have come from.

By simply authenticating to the registry to push while a strong first defense introduces further requirements for protecting those credentials and particularly when your CI system is creating the images those credentials may be visible to multiple team members.

This is then exasperated if you’re running your own registry since the administrators of that are likely administrators of other things, which lowers the barrier for how much collusion is required (too much privilege – trust is extended a long way).

Let’s play a game of ‘spot the difference’: One of these images has something malicious in it that’ll do bad things when it runs (not really, it’ll just print “I am the bad guy” and then exit, but you get the point). The table shows the output of the Continuous Integration pipeline in GitHub and the corresponding images it built and pushed, storing it in Github Container Registry.

Build #7

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1

Unsiged Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417835412.1

Build #8

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417836769.1

Build #9

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417837856.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417837856.1

Build #10

Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417838926.1

Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417838926.1

Spot anything by just looking at the unsigned images? 

By looking at the metadata of the build outputs it is hard to tell if any of these images have been changed since they were built from the CI pipeline. Spoiler alert: It’s Build #7.

Okay, maybe I made this one easy since the time it took me to write the bad code, push and sign it under my own signature makes it stand out in the registry.

One way of checking if the image has been tampered with is to use a utility called cosign. If we take the signed image name from Build #8 and run the following command:

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1 | jq

The following output is shown:

"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-demo-spotthedifference/.github/workflows/ci.yml@refs/heads/main"

If we run the same command but this time with the image from Build #7, you can see that the Issuer and subject are different. 

$COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1 | jq
      "Issuer": "https://github.com/login/oauth",
      "Subject": "chris@cns.me.uk"

You can verify this yourself by installing cosign and running against any of the images in our demonstration repo. In the two outputs above, it can be seen image in Build #8 was created by a pipeline and the image from Build #7 was created by a bad actor aka chris@cns.me.uk.

Let’s not forget that if a bad actor has write access to your registry (or the ability to impersonate it) then they don’t need to leave their bad image there for long. They can always replace it with the good one later after they’ve reached their goal of it running in your cluster; hence while signing is important, constantly verifying that is how you unlock the value.

Traditional issues with keypair signing

When creating and pushing signed images to a container registry, two things are required: registry credentials and a private key. You’ve now got two problems: Securing registry credentials and securing a private key. Inevitably you’ll keep these in a similar way, a compromised device such as a developer laptop or CI system quickly negates any of the benefits of signing in the first place.

It’s possible to use hardware tokens such as a Yubikey for humans which can be configured to require a PIN and physically touch the token to sign anything. It is not possible to extract the private key material from the device. However, this is inherently not practical to implement for your non-human fingerless builders.

Enter: Cosign

Cosign from sigstore makes the traditional signing of container images vastly easier. However, this can be taken a step further in combination with the wider sigstore products. Using fulcio and rekor, it’s possible to sign images (and other things too) without keys and later verify them against an immutable transparency log.

So, how do you sign without keys?

Okay, we don’t really sign without keys. But behind the scenes, cosign creates the keypair ephemerally (they last 20 minutes) and gets them signed by Fulcio using your authenticated OIDC identity.


OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

keyless signing

OIDC with human interaction

$ COSIGN_EXPERIMENTAL=1 cosign sign chrisns/cosign-keyless-demo:latest

$ COSIGN_EXPERIMENTAL=1 cosign verify chrisns/cosign-keyless-demo:latest

Note the issuer and subject, this is from the identity Fulcio received and stored in the Rekor transparency log.

"Issuer": "https://accounts.google.com",
"Subject": "chris.nesbitt-smith@appvia.io"

OIDC flow with interaction-free with GitHub actions

Now lets do it without interaction, using the following workflow:

name: Build Push Sign
on: { push: { branches: ['main'] } }

    runs-on: ubuntu-latest
      packages: write
      id-token: write

      - uses: actions/checkout@v2.3.5

      - name: Login to GitHub
        uses: docker/login-action@v1.9.0
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: build+push
        uses: docker/build-push-action@v2.7.0
          push: true
          tags: ghcr.io/chrisns/cosign-keyless-demo:latest

      - uses: sigstore/cosign-installer@main

      - name: Sign the images
        run: |
          cosign sign ghcr.io/chrisns/cosign-keyless-demo:latest

Now your images will have something like the following when you run:

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-keyless-demo:latest
"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main"

What this allows us to do is cryptographically demonstrate that the image we can pull was built by the CI pipeline of that repository. Neat!

Who should sign their containers?

As we’ve seen, signing your container images can be as trivial as a couple of lines added to your CI pipeline.

If you’re a software vendor that ships their product as a container, signing is a great way to reassure your customers that you take security seriously and allow them to verify the authenticity of your product.

If you’re building software for your own business use, it’s also a great way to validate the integrity of your own supply chain internally.

Verifying the OIDC signed image

So now that we can verify the image manually, the next logical step is to let our Kubernetes cluster do that for us.

Kubernetes admission controller

In Kubernetes, an Admissions Controller is a piece of code that intercepts requests to the API server and does something with it. We have created an admissions controller that can either approve or deny the image from running in the cluster based on its cryptographic signature. 

For the next step, you’ll need a Kubernetes cluster. If you don’t have one to hand, you can use KiND or minikube.


# if you don't already have cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -k https://github.com/appvia/cosign-keyless-admission-webhook


In your pod spec you set an annotation(s) of subject.cosign.sigstore.dev/CONTAINERNAME* to the subject of the certificate and also set the issuer.cosign.sigstore.dev/CONTAINERNAME* to the Issuer.

*CONTAINER_NAME is the name of the container from your pod specification.

Full example:

apiVersion: v1
kind: Pod
    subject.cosign.sigstore.dev/demo: https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main
    issuer.cosign.sigstore.dev/demo: https://token.actions.githubusercontent.com
  name: cosign-keyless-demo
    - image: ghcr.io/chrisns/cosign-keyless-demo:latest
      name: demo

Save the contents above in a file called pod.yaml and run: 

$ kubectl apply -f pod.yaml

Since the image ghcr.io/chrisns/cosign-keyless-demo:latest is signed as we expect the Admission controller to accept the request and the pod will be accepted.

And if you change the pod.yaml to:

apiVersion: v1
kind: Pod
    subject.cosign.sigstore.dev/demo: chris@cns.me.uk
    issuer.cosign.sigstore.dev/demo: "https://github.com/login/oauth",
  name: cosign-keyless-demo-bad
    - image: ghcr.io/chrisns/cosign-keyless-demo:latest
      name: demo

You’ll find it will be rejected by the admission controller and not permitted to run on the cluster as this image is not signed as expected.

Known limitations and potential improvements

  • Untested with registries that require authentication
  • No caching, so it checks each pod, which can be slow
  • Requires connectivity to the transparency log and the registry
  • Wildcards and more complex rules are defined on the namespace or potentially cluster level
  • Doesn’t check signatures on initContainers

Start signing your images

If you’ve followed along with the above you can now sign your images, and check the signatures in Kubernetes before accepting the pod to run, assuring that the image your cluster is about to start hasn’t been tampered with. Ensuring that the images form part of a secure software supply chain from creation to runtime.

The admission controller I made is open-source on GitHub, so please do star and watch it, and as always pull requests are always very welcome.

About Appvia

Appvia enables businesses to solve complex cloud challenges with products and services that make Kubernetes secure, cost-effective and scalable.

Our founders have worked with Kubernetes in highly regulated, highly secure environments since 2016, contributing heavily to innovative projects such as Kops and fully utilizing Kubernetes ahead of the curve. We’ve mastered Kubernetes, and experienced its complexities, so our customers don’t have to. 

Share this article
profile-112x112-crop-1 (4)
Chris Nesbitt-Smith
A developer at heart that’s now more focused on people and less on the technical implementation detail. Although, when I’m not building stuff with the kids, I spend time on open source work and co-maintain a few high-profile repositories in the home automation space.

The podcast that takes a lighthearted look at the who, what, when, where, why, how and OMGs of cloud computing

Related insights

Managing Kubernetes Secrets with HashiCorp Vault vs. Azure Key Vault Keeping secrets secure...
Namespaces are a vital feature of Kubernetes. They allow you to separate uniquely named...
DevOps teams have rapidly adopted Kubernetes as the standard way to deploy and...
Once you start working with Kubernetes, it’s natural to think about how you...
Self-service of cloud resources Kubernetes has been brilliant at delivering an ecosystem for...
Pods, deployments, and services are just some of the concepts that you need to understand in...
Last week I published a blog, “How to spot gaps in your Public Cloud...
Breaking down the core areas that you should be aware of when considering...
5 tips to help you manage more with less Not every manager of...
Public cloud has provided huge benefits in getting infrastructure and services to people...
This is the story of how three Appvia Engineers contributed so much to...
Overview The UK Home Office is a large government organisation, whose projects and...