Overview
Cosign is a command line utility that can sign and verify software artifact, such as container images and blobs.
In Kubernetes, we can use FluxCD and Kyverno to verify Helmcharts and Docker images respectively.
Image Added
Installation
https://docs.sigstore.dev/system_config/installation/
...
Code Block |
---|
Enter password for private key:
Enter password for private key again:
Private key written to cosign.key
Public key written to cosign.pub |
Signing and Verifying a Container
Get the digest of the image
...
Code Block |
---|
Enter password for private key:
WARNING: "ncydacrinprogress.azurecr.io/cybersecuritydome/kafka-azure-sink" appears to be a private repository, please confirm uploading to the transparency log at "https://rekor.sigstore.dev"
Are you sure you would like to continue? [y/N] y
The sigstore service, hosted by sigstore a Series of LF Projects, LLC, is provided pursuant to the Hosted Project Tools Terms of Use, available at https://lfprojects.org/policies/hosted-project-tools-terms-of-use/.
Note that if your submission includes personal data associated with this signed artifact, it will be part of an immutable record.
This may include the email address associated with the account with which you authenticate your contractual Agreement.
This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later, and is subject to the Immutable Record notice at https://lfprojects.org/policies/hosted-project-tools-immutable-records/.
By typing 'y', you attest that (1) you are not submitting the personal data of any other person; and (2) you understand and agree to the statement and the Agreement terms at the URLs listed above.
Are you sure you would like to continue? [y/N] y
tlog entry created with index: 77828277
Pushing signature to: ncydacrinprogress.azurecr.io/cybersecuritydome/kafka-azure-sink |
References
View Container in Azure
Image Added
Verify Signature
Code Block |
---|
|
cosign verify --key cosign.pub ncydacrinprogress.azurecr.io/cybersecuritydome/kafka-azure-sink@sha256:2fbb556a6a2b68466def031067c18411693c6a9f3b5e4b16c1677e28c0029172 |
Code Block |
---|
Verification for ncydacrinprogress.azurecr.io/cybersecuritydome/kafka-azure-sink@sha256:2fbb556a6a2b68466def031067c18411693c6a9f3b5e4b16c1677e28c0029172 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The signatures were verified against the specified public key |
Code Block |
---|
|
[
{
"critical": {
"identity": {
"docker-reference": "ncydacrinprogress.azurecr.io/cybersecuritydome/kafka-azure-sink"
},
"image": {
"docker-manifest-digest": "sha256:2fbb556a6a2b68466def031067c18411693c6a9f3b5e4b16c1677e28c0029172"
},
"type": "cosign container image signature"
},
"optional": {
"Bundle": {
"SignedEntryTimestamp": "MEUCIQCt8OCvcaUzKeee109JVEOTOx+2DEKA5SCEd5R/BaXA1QIgI91Ebfv6MEx5F2OW05yU8kSxS3kwrXAP5/beU7CQBc0=",
"Payload": {
"body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiI5MTMyMWRiYjc3YWM4NzI4ZmE4ZmUwNDZmNjgxMmNlMDg5NjQ1MDZjZDU5Y2UxNTk4MzMyM2EwNDU2NzE5OWFmIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJUUNEYkR3MU9PSkZLTVRrYWJrN0J2Umw3NHVNVE54TXdnM0NkdzNZWVUxRTVnSWdINWp1UmJRd2tnSnozZEZJNkEreFIxc2tnV2N3NmFsTUdublFKaU9PNC80PSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGTmpnNE56a3pPVlZtVkRsUFVFMUlkbE5VTjA5Q1psUXhlRUYyWVFwcFVsQmlRakZJZVdGeUsyNUdRMVZYVm5aWU4wVjJhVVZRVEhoVVdsSk9VVEpCTkU5UVMwRnJSRzh4WlROSVNUaFBSbFJ5T1ZwQlNYbFJQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=",
"integratedTime": 1710344087,
"logIndex": 77828277,
"logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
}
}
}
}
] |
Signing and Verifying a Helm Chart
Assuming you have pushed a helm chart to your oci repository
Code Block |
---|
helm push <app-name>-<app-version>.tgz oci://<registry-host>/<org>/charts |
Sign
Code Block |
---|
cosign sign --key cosign.key <registry-host>/<org>/charts/<app-name>:<app-version>
ex:
cosign sign --key cosign.key ncydacrinprogress.azurecr.io/charts/kowl:22.0.1-4040670 |
Add the public key to the cluster
Code Block |
---|
kubectl -n flux-system create secret generic cosign-pub --from-file=cosign.pub=cosign.pub
ex:
cd ~/cosign
kubectl -n ncyd-flux create secret generic cosign-pub --from-file=cosign.pub=cosign.pub
|
Modify helmrelease to verify the helmchart
Code Block |
---|
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: <app-name>
spec:
interval: 1h
chart:
spec:
chart: <app-name>
version: <app-version>
sourceRef:
kind: HelmRepository
name: helm-charts
verify:
provider: cosign
secretRef:
name: cosign-pub |
When using a customization override
Code Block |
---|
language | yml |
---|
title | kowl.yaml |
---|
|
---
# set $patch: delete to exclude from installation
#$patch: delete
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: kowl
namespace: ncyd-flux
spec:
chart:
spec:
version: '22.0.1-4040670'
verify:
provider: cosign
secretRef:
name: cosign-pub
values:
imagePullSecrets:
- name: regcred
image:
registry: ncydacrinprogress.azurecr.io
|
Reconcile and watch for events in the namespace
Code Block |
---|
flux reconcile kustomization ncyd-flux -n ncyd-flux
kubectl events -n ncyd-flux --watch |
Error from source-controller logs:
Code Block |
---|
chart verification error: not implemented
{"level":"error","ts":"2024-03-14T15:19:53.977Z","msg":"Reconciler error","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"ncyd-flux-kowl","namespace":"ncyd-flux"},"namespace":"ncyd-flux","name":"ncyd-flux-kowl","reconcileID":"d103f844-f750-49cf-ba56-a631037fd585","error":"chart verification error: not implemented"} |
I think we need to define an oci repository. Currently we have a helm repository.
Definen OCI Repository
Use it..
Code Block |
---|
language | yml |
---|
title | kowl.yaml |
---|
|
---
# set $patch: delete to exclude from installation
#$patch: delete
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: kowl
namespace: ncyd-flux
spec:
chart:
spec:
version: '22.0.1-4040670'
sourceRef:
kind: HelmRepository
name: ncyd-oci-virtual
verify:
provider: cosign
secretRef:
name: cosign-pub
values:
imagePullSecrets:
- name: regcred
image:
registry: ncydacrinprogress.azurecr.io
|
If the signature is bad
Code Block |
---|
> kubectl events -n ncyd-flux --watch
0s (x4 over 20s) Warning ChartVerificationError HelmChart/ncyd-flux-kowl chart verification error: failed to verify oci://ncydacrinprogress.azurecr.io/charts/kowl:22.0.1-4042072: no matching signatures: invalid signature when validating ASN.1 encoded signature
> f logs -f source-controller-f7dbc4597-vwwj2
{"level":"error","ts":"2024-03-15T16:08:05.400Z","msg":"Reconciler error","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"ncyd-flux-kowl","namespace":"ncyd-flux"},"namespace":"ncyd-flux","name":"ncyd-flux-kowl","reconcileID":"c29c0b58-cf10-4d1d-a290-ecb997acf1ac","error":"chart verification error: failed to verify oci://ncydacrinprogress.azurecr.io/charts/kowl:22.0.1-4042072: no matching signatures: invalid signature when validating ASN.1 encoded signature"} |
Process for Using Cosign with HelmCharts in Flux
- Need to generate keys
Code Block |
---|
|
cosign generate-key-pair |
- Need to store public key in k8s
Code Block |
---|
kubectl -n flux-system create secret generic cosign-pub --from-file=cosign.pub=cosign.pub
ex:
cd ~/cosign
kubectl -n ncyd-flux create secret generic cosign-pub --from-file=cosign.pub=cosign.pub
|
- Need to use OCI helmrepository
Code Block |
---|
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: ncyd-oci-virtual
namespace: ncyd-flux
spec:
type: "oci"
interval: 1m0s
secretRef:
name: regcred
url: oci://ncydacrinprogress.azurecr.io/charts |
- Need to sign helmchart
Code Block |
---|
cosign sign --key cosign.key <registry-host>/<org>/charts/<app-name>:<app-version>
ex:
cosign sign --key cosign.key ncydacrinprogress.azurecr.io/charts/kowl:22.0.1-4040670 |
- Need to update helm releases to use OCI and reference cosign public key
Code Block |
---|
language | yml |
---|
title | kowl.yaml |
---|
|
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: kowl
namespace: ncyd-flux
spec:
chart:
spec:
sourceRef:
kind: HelmRepository
name: ncyd-oci-virtual
verify:
provider: cosign
secretRef:
name: cosign-pub
values:
...
|
Signing and Verifying Images
https://fluxcd.io/blog/2022/02/security-image-provenance/
https://kyverno.io/docs/writing-policies/verify-images/
Install Kyverno
Code Block |
---|
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install -n ncyd-flux kyverno kyverno/kyverno
|
Update the kyverno deployment
Code Block |
---|
f edit deployment/kyverno-admission-controller |
Code Block |
---|
containers:
- args:
...
- --imagePullSecrets=regcred
|
Create Policy:
Code Block |
---|
apiVersion: kyverno.io/v1
kind: Policy
metadata:
name: check-image
spec:
validationFailureAction: Enforce
background: false
webhookTimeoutSeconds: 30
failurePolicy: Fail
rules:
- name: check-image
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ncydacrinprogress.azurecr.io/cloudhut/kowl:*"
attestors:
- count: 1
entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6887939UfT9OPMHvST7OBfT1xAva
iRPbB1Hyar+nFCUWVvX7EviEPLxTZRNQ2A4OPKAkDo1e3HI8OFTr9ZAIyQ==
-----END PUBLIC KEY-----
|
https://kyverno.io/docs/writing-policies/verify-images/sigstore/
Note
The public key may either be defined in the policy directly or reference a standard Kubernetes Secret elsewhere in the cluster by specifying it in the format k8s://<namespace>/<secret_name>
. The named Secret must specify a key cosign.pub
containing the public key used for verification. Secrets may also be referenced using the secret{}
object. See kubectl explain clusterpolicy.spec.rules.verifyImages.attestors.entries.keys
for more details on the supported key options.
Testing
Code Block |
---|
> kubectl events -n ncyd-flux --watch
0s Warning error HelmRelease/kowl Helm install failed: 1 error occurred:
* admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Deployment/ncyd/kowl was blocked due to the following policies
check-image:
autogen-check-image: 'failed to verify image ncydacrinprogress.azurecr.io/cloudhut/kowl:v1.5.0:
.attestors[0].entries[0].keys: GET https://ncydacrinprogress.azurecr.io/oauth2/token?scope=repository%!A(MISSING)cloudhut%!F(MISSING)kowl%!A(MISSING)pull&service=ncydacrinprogress.azurecr.io:
UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization
for more information.'
Last Helm logs:
0s Warning error HelmRelease/kowl reconciliation failed: Helm install failed: 1 error occurred:
* admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Deployment/ncyd/kowl was blocked due to the following policies
check-image:
autogen-check-image: 'failed to verify image ncydacrinprogress.azurecr.io/cloudhut/kowl:v1.5.0:
.attestors[0].entries[0].keys: GET https://ncydacrinprogress.azurecr.io/oauth2/token?scope=repository%!A(MISSING)cloudhut%!F(MISSING)kowl%!A(MISSING)pull&service=ncydacrinprogress.azurecr.io:
UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization
for more information.' |
Script to Sign Helm Charts and Docker Images
Code Block |
---|
language | bash |
---|
title | signArtifacts.sh |
---|
|
#!/bin/bash
#source optional env file
set -a
. ~/cosign/env
set -e +a
#************ SAMPLE ENV FILE ************
#BUILD="22.0.1-4040670"
#SRE_BUILD="23.2.0-3610795"
#OCI_REPO="xxx.azurecr.io"
#OCI_REPO_USERNAME=xxx
#OCI_REPO_PASSWORD="xxx"
#COSIGN_PRIVATE_KEY="-----BEGIN ENCRYPTED SIGSTORE PRIVATE KEY-----
#...
#-----END ENCRYPTED SIGSTORE PRIVATE KEY-----"
#COSIGN_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----
#...
#-----END PUBLIC KEY-----"
export COSIGN_PASSWORD=""
signArtifact () {
cosign sign --yes --key env://COSIGN_PRIVATE_KEY \
--registry-username="${OCI_REPO_USERNAME}" \
--registry-password="${OCI_REPO_PASSWORD}" \
${OCI_REPO}/$1
}
verifyArtifact(){
cosign verify --key env://COSIGN_PUBLIC_KEY \
--registry-username="${OCI_REPO_USERNAME}" \
--registry-password="${OCI_REPO_PASSWORD}" \
${OCI_REPO}/$1 >/dev/null
}
source images.src
source charts.src
images=(${images[@]})
charts=(${charts[@]})
requiredBins=(
"cosign"
)
for bin in "${requiredBins[@]}"; do
if ! command -v ${bin} &> /dev/null
then
echo "Required command ${bin} could not be found, please install and re-run"
exit
fi
done
echo
echo "=================="
echo "SIGNING CHARTS"
echo "=================="
for chart in "${charts[@]}"; do
chartName=`echo ${chart}|sed 's/:.*//'`
chartVersion=`echo ${chart}|sed -e 's/.*://'`
echo "ChartName: ${chartName}"
echo "ChartVersion: ${chartVersion}"
echo "------------------"
signArtifact charts/${chartName}:${chartVersion}
verifyArtifact charts/${chartName}:${chartVersion}
echo "------------------"
done
echo
echo "=================="
echo "SIGNING IMAGES"
echo "=================="
for image in "${images[@]}"; do
imageNameAndVersion=`echo $image |sed 's:[^/]*/\(.*\):\1:'`
imageName=`echo ${imageNameAndVersion}|sed -e 's/:.*//'`
imageVersion=`echo ${imageNameAndVersion}|sed -e 's/.*://'`
echo "Image Name: ${imageName}"
echo "Image Version: ${imageVersion}"
echo "------------------"
signArtifact ${imageName}:${imageVersion}
verifyArtifact ${imageName}:${imageVersion}
echo "------------------"
done
echo ""
echo "DONE!"
|
Code Block |
---|
|
images=(
pkg/fluent-bit:$BUILD"
)
|
Code Block |
---|
|
charts=(
"mychart:$BUILD"
"ckaf/kafka/rocky8:8.4.2-7.3.1-7486"
) |
Helpful Scripts
Install latest helm
Code Block |
---|
https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash |
Login to Helm Registry
Code Block |
---|
helm registry login ncydacrinprogress.azurecr.io --username ncydacrinprogress --password xxx |
Push Chart
Code Block |
---|
helm push ssh-server-0.1.0.tgz oci://ncydacrinprogress.azurecr.io/charts |
List artifacts in Azure Container Registry
Code Block |
---|
az acr login --name ncydacrinprogress.azurecr.io -u ncydacrinprogress -p xxxx
az acr repository list --name ncydacrinprogress.azurecr.io -u ncydacrinprogress -p xxxx -o tsv
|
Output
Code Block |
---|
acm/agent
acm/block-provider
acm/frontend
acm/maria-db
acm/mockserver
acm/server
atlassian/jira-software
azuremonitor/containerinsights/ciprod
bats/bats
bitnami/nginx
... |
Using ORAS
https://oras.land
Install
Code Block |
---|
brew install oras |
Login
Code Block |
---|
oras login ${OCI_REPO} --username ${OCI_REPO_USERNAME} --password ${OCI_REPO_PASSWORD} |
List Artifacts
Code Block |
---|
oras repo ls ${OCI_REPO}/ |
Get Latest Tag
Code Block |
---|
oras repo tags ${OCI_REPO}/${artifact} | tail -n 1 |
References