How to set this up was not really well documented so this post is an attempt at that.
First we will look at how to setup a GitHub workflow using GitHub Actions to build multi-architecture container images with buildah and push them to a registry with podman. Then we will sign those images with cosign (sigstore) and detail what is needed to configure signature validation on the host. Finally we will detail the remaining work needed to be able to do the entire process only with podman.
Full example ready to go
If you just want to get going, you can copy the content of my github.com/travier/cosign-test repo and start building and pushing your containers. I recommend keeping only the
cosign.yaml workflow for now (see below for the details).
“Minimal” GitHub workflow to build containers with buildah / podman
You can find those actions at github.com/redhat-actions.
Here is an example workflow with the Containerfile in the
example sub directory:
This should let you to test changes to the image via builds in pull requests and publishing the changes only once they are merged.
You will have to setup the
BOT_SECRET secrets in the repository configuration to push to the registry of your choice.
If you prefer to use the GitHub internal registry then you can use:
You will also need to set the job permissions to be able to write GitHub Packages (container registry):
See the Publishing Docker images GitHub Docs.
You should also configure the GitHub Actions settings as follow:
- In the “Actions permissions” section, you can restict allowed actions to: “Allow <username>, and select non-<username>, actions and reusable workflows”, with “Allow actions created by GitHub” selected and the following additionnal actions:
In the “Workflow permissions” section, you can select the “Read repository contents and packages permissions” and select the “Allow GitHub Actions to create and approve pull requests”.
- Make sure to add all the required secrets in the “Secrets and variables”, “Actions”, “Repository secrets” section.
Signing container images
We will use cosign to sign container images. With cosign, you get two main options to sign your containers:
- Keyless signing: Sign containers with ephemeral keys by authenticating with an OIDC (OpenID Connect) protocol supported by Sigstore.
- Self managed keys: Generate a “classic” long-lived key pair.
We will choose the the “self managed keys” option here as it is easier to setup for verification on the host in podman. I will likely make another post once I figure out how to setup keyless signature verification in podman.
Generate a key pair with:
Enter an empty password as we will store this key in plain text as a repository secret (
Then you can add the steps for signing with cosign at the end of your workflow:
2024-01-12 update: Sign container images recursively for multi-arch images.
We need to explicitly login to the container registry to get an auth token that will be used by cosign to push the signature to the registry.
This step sometimes fails, likely due to a race condition, that I have not been able to figure out yet. Retrying failed jobs usually works.
You should then update the GitHub Actions settings to allow the new actions as follows:
Configuring podman on the host to verify image signatures
First, we copy the public key to a designated place in
Then we setup the registry config to tell it to use sigstore signatures:
And then we update the container signature verification policy to:
- Default to reject everything
- Then for the docker transport:
- Verify signatures for containers coming from our repository
- Accept all other containers from other registries
If you do not plan on using container from other registries, you can even be stricter here and only allow your containers to be used.
See the full man page for containers-policy.json(5).
You should now be good to go!
What about doing everything with podman?
Using this workflow, there is a (small) time window where the container images are pushed to the registry but not signed.
One option to avoid this problem would be to first push the container to a “temporary” tag first, sign it, and then copy the signed container to the latest tag.
Another option is to use podman to push and sign the container image “at the same time”. However podman still needs to push the image first and then sign it so there is still a possibility that signing fails and that you’re left with an unsigned image (this happened to me during testing).
Unfortunately for us, the version of podman available in the version of Ubuntu used for the GitHub Runners (22.04) is too old to support signing containers. We thus need to use a newer podman from a container image to workaround this.
Here is the same workflow, adapted to only use podman for signing:
This uses two additional workarounds for missing features:
- There is no official container image that includes both podman and buildah right now, thus I made one: github.com/travier/podman-action
redhat-actions/push-to-registryAction does not support signing yet (issue#89). I’ve implemented support for self managed key signing in pull#90. I’ve not looked at keyless signing yet.
You will also have to allow running my actions in the repository settings. In the “Actions permissions” section, you should use the following actions:
The next steps are to figure out all the missing bits for keyless signing and replicate this entire process in GitLab CI.