Signed container images with buildah, podman and cosign via GitHub Actions

All the Toolbx and Distrobox container images and the ones in my personal namespace on Quay.io are now signed using cosign.

How to set this up was not really well documented so this post is an attempt at that.

First we will look at how to setup a GitHub workflow using GitHub Actions to build multi-architecture container images with buildah and push them to a registry with podman. Then we will sign those images with cosign (sigstore) and detail what is needed to configure signature validation on the host. Finally we will detail the remaining work needed to be able to do the entire process only with podman.

Full example ready to go

If you just want to get going, you can copy the content of my github.com/travier/cosign-test repo and start building and pushing your containers. I recommend keeping only the cosign.yaml workflow for now (see below for the details).

“Minimal” GitHub workflow to build containers with buildah / podman

You can find those actions at github.com/redhat-actions.

Here is an example workflow with the Containerfile in the example sub directory:

name: "Build container using buildah/podman"

env:
  NAME: "example"
  REGISTRY: "quay.io/example"

on:
  # Trigger for pull requests to the main branch, only for relevant files
  pull_request:
    branches:
      - main
    paths:
      - 'example/**'
      - '.github/workflows/cosign.yml'
  # Trigger for push/merges to main branch, only for relevant files
  push:
    branches:
      - main
    paths:
      - 'example/**'
      - '.github/workflows/cosign.yml'
  # Trigger every Monday morning
  schedule:
    - cron:  '0 0 * * MON'

permissions: read-all

# Prevent multiple workflow runs from racing to ensure that pushes are made
# sequentialy for the main branch. Also cancel in progress workflow runs for
# pull requests only.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  build-push-image:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: Setup QEMU for multi-arch builds
        shell: bash
        run: |
          sudo apt install qemu-user-static

      - name: Build container image
        uses: redhat-actions/buildah-build@v2
        with:
          # Only select the architectures that matter to you here
          archs: amd64, arm64, ppc64le, s390x
          context: ${{ env.NAME }}
          image: ${{ env.NAME }}
          tags: latest
          containerfiles: ${{ env.NAME }}/Containerfile
          layers: false
          oci: true

      - name: Push to Container Registry
        uses: redhat-actions/push-to-registry@v2
        # The id is unused right now, will be used in the next steps
        id: push
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'
        with:
          username: ${{ secrets.BOT_USERNAME }}
          password: ${{ secrets.BOT_SECRET }}
          image: ${{ env.NAME }}
          registry: ${{ env.REGISTRY }}
          tags: latest

This should let you to test changes to the image via builds in pull requests and publishing the changes only once they are merged.

You will have to setup the BOT_USERNAME and BOT_SECRET secrets in the repository configuration to push to the registry of your choice.

If you prefer to use the GitHub internal registry then you can use:

env:
  REGISTRY: ghcr.io/${{ github.repository_owner }}

...
  username: ${{ github.actor }}
  password: ${{ secrets.GITHUB_TOKEN }}

You will also need to set the job permissions to be able to write GitHub Packages (container registry):

permissions:
  contents: read
  packages: write

See the Publishing Docker images GitHub Docs.

You should also configure the GitHub Actions settings as follow:

• In the “Actions permissions” section, you can restict allowed actions to: “Allow <username>, and select non-<username>, actions and reusable workflows”, with “Allow actions created by GitHub” selected and the following additionnal actions:

  redhat-actions/*,
  

• In the “Workflow permissions” section, you can select the “Read repository contents and packages permissions” and select the “Allow GitHub Actions to create and approve pull requests”.

• Make sure to add all the required secrets in the “Secrets and variables”, “Actions”, “Repository secrets” section.

Signing container images

We will use cosign to sign container images. With cosign, you get two main options to sign your containers:

• Keyless signing: Sign containers with ephemeral keys by authenticating with an OIDC (OpenID Connect) protocol supported by Sigstore.
• Self managed keys: Generate a “classic” long-lived key pair.

We will choose the the “self managed keys” option here as it is easier to setup for verification on the host in podman. I will likely make another post once I figure out how to setup keyless signature verification in podman.

Generate a key pair with:

$ cosign generate-key-pair

Enter an empty password as we will store this key in plain text as a repository secret (COSIGN_PRIVATE_KEY).

Then you can add the steps for signing with cosign at the end of your workflow:

      # Include at the end of the workflow previously defined

      - name: Login to Container Registry
        uses: redhat-actions/podman-login@v1
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.BOT_USERNAME }}
          password: ${{ secrets.BOT_SECRET }}

      - uses: sigstore/cosign-installer@v3.3.0
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'

      - name: Sign container image
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'
        run: |
          cosign sign -y --recursive --key env://COSIGN_PRIVATE_KEY ${{ env.REGISTRY }}/${{ env.NAME }}@${{ steps.push.outputs.digest }}
        env:
          COSIGN_EXPERIMENTAL: false
          COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}

2024-01-12 update: Sign container images recursively for multi-arch images.

We need to explicitly login to the container registry to get an auth token that will be used by cosign to push the signature to the registry.

This step sometimes fails, likely due to a race condition, that I have not been able to figure out yet. Retrying failed jobs usually works.

You should then update the GitHub Actions settings to allow the new actions as follows:

redhat-actions/*,
sigstore/cosign-installer@*,

Configuring podman on the host to verify image signatures

First, we copy the public key to a designated place in /etc:

$ sudo mkdir /etc/pki/containers
$ curl -O "https://.../cosign.pub"
$ sudo cp cosign.pub /etc/pki/containers/
$ sudo restorecon -RFv /etc/pki/containers

Then we setup the registry config to tell it to use sigstore signatures:

$ cat /etc/containers/registries.d/quay.io-example.yaml
docker:
  quay.io/example:
    use-sigstore-attachments: true
$ sudo restorecon -RFv /etc/containers/registries.d/quay.io-example.yaml

And then we update the container signature verification policy to:

• Default to reject everything
• Then for the docker transport:
  • Verify signatures for containers coming from our repository
  • Accept all other containers from other registries

If you do not plan on using container from other registries, you can even be stricter here and only allow your containers to be used.

/etc/containers/policy.json:

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            ...
            "quay.io/example": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/etc/pki/containers/quay.io-example.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ],
            ...
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        },
        ...
    }
}

See the full man page for containers-policy.json(5).

You should now be good to go!

What about doing everything with podman?

Using this workflow, there is a (small) time window where the container images are pushed to the registry but not signed.

One option to avoid this problem would be to first push the container to a “temporary” tag first, sign it, and then copy the signed container to the latest tag.

Another option is to use podman to push and sign the container image “at the same time”. However podman still needs to push the image first and then sign it so there is still a possibility that signing fails and that you’re left with an unsigned image (this happened to me during testing).

Unfortunately for us, the version of podman available in the version of Ubuntu used for the GitHub Runners (22.04) is too old to support signing containers. We thus need to use a newer podman from a container image to workaround this.

Here is the same workflow, adapted to only use podman for signing:

name: "Build container using buildah, push and sign it using podman"

env:
  NAME: "example"
  REGISTRY: "quay.io/example"
  REGISTRY_DOMAIN: "quay.io"

on:
  pull_request:
    branches:
      - main
    paths:
      - 'example/**'
      - '.github/workflows/podman.yml'
  push:
    branches:
      - main
    paths:
      - 'example/**'
      - '.github/workflows/podman.yml'
  schedule:
    - cron:  '0 0 * * MON'

permissions: read-all

# Prevent multiple workflow runs from racing to ensure that pushes are made
# sequentialy for the main branch. Also cancel in progress workflow runs for
# pull requests only.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  build-push-image:
    runs-on: ubuntu-latest
    container:
      image: quay.io/travier/podman-action
      options: --privileged -v /proc/:/host/proc/:ro
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: Setup QEMU for multi-arch builds
        shell: bash
        run: |
          for f in /usr/lib/binfmt.d/*; do cat $f | sudo tee /host/proc/sys/fs/binfmt_misc/register; done
          ls /host/proc/sys/fs/binfmt_misc

      - name: Build container image
        uses: redhat-actions/buildah-build@v2
        with:
          archs: amd64, arm64, ppc64le, s390x
          context: ${{ env.NAME }}
          image: ${{ env.NAME }}
          tags: latest
          containerfiles: ${{ env.NAME }}/Containerfile
          layers: false
          oci: true

      - name: Setup config to enable pushing Sigstore signatures
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'
        shell: bash
        run: |
          echo -e "docker:\n  ${{ env.REGISTRY_DOMAIN }}:\n    use-sigstore-attachments: true" \
            | sudo tee -a /etc/containers/registries.d/${{ env.REGISTRY_DOMAIN }}.yaml

      - name: Push to Container Registry
        # uses: redhat-actions/push-to-registry@v2
        uses: travier/push-to-registry@sigstore-signing
        if: (github.event_name == 'push' || github.event_name == 'schedule') && github.ref == 'refs/heads/main'
        with:
          username: ${{ secrets.BOT_USERNAME }}
          password: ${{ secrets.BOT_SECRET }}
          image: ${{ env.NAME }}

This uses two additional workarounds for missing features:

• There is no official container image that includes both podman and buildah right now, thus I made one: github.com/travier/podman-action
• The redhat-actions/push-to-registry Action does not support signing yet (issue#89). I’ve implemented support for self managed key signing in pull#90. I’ve not looked at keyless signing yet.

You will also have to allow running my actions in the repository settings. In the “Actions permissions” section, you should use the following actions:

redhat-actions/*,
travier/push-to-registry@*,

Conclusion

The next steps are to figure out all the missing bits for keyless signing and replicate this entire process in GitLab CI.