...
Cosign is a command line utility that can sign and verify software artifact, such as container images and blobs.
In Kubernetes, we can use FluxCD and Kyverno to verify Helmcharts and Docker images respectively.
Installation
https://docs.sigstore.dev/system_config/installation/
...
Code Block | ||||
---|---|---|---|---|
| ||||
apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: kowl namespace: ncyd-flux spec: chart: spec: sourceRef: kind: HelmRepository name: ncyd-oci-virtual verify: provider: cosign secretRef: name: cosign-pub values: ... |
References
...
Signing and Verifying Images
...
...
fluxcd.io/blog/2022/02/security-image-provenance/
...
...
docs/writing-policies/verify-images/
Install Kyverno
Code Block |
---|
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install -n ncyd-flux kyverno kyverno/kyverno
|
Update the kyverno deployment
Code Block |
---|
f edit deployment/kyverno-admission-controller |
Code Block |
---|
containers:
- args:
...
- --imagePullSecrets=regcred
|
Create Policy:
Code Block |
---|
apiVersion: kyverno.io/v1
kind: Policy
metadata:
name: check-image
spec:
validationFailureAction: Enforce
background: false
webhookTimeoutSeconds: 30
failurePolicy: Fail
rules:
- name: check-image
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ncydacrinprogress.azurecr.io/cloudhut/kowl:*"
attestors:
- count: 1
entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE6887939UfT9OPMHvST7OBfT1xAva
iRPbB1Hyar+nFCUWVvX7EviEPLxTZRNQ2A4OPKAkDo1e3HI8OFTr9ZAIyQ==
-----END PUBLIC KEY-----
|
https://kyverno.io/docs/writing-policies/verify-images/sigstore/
Note
The public key may either be defined in the policy directly or reference a standard Kubernetes Secret elsewhere in the cluster by specifying it in the format
k8s://<namespace>/<secret_name>
. The named Secret must specify a keycosign.pub
containing the public key used for verification. Secrets may also be referenced using thesecret{}
object. Seekubectl explain clusterpolicy.spec.rules.verifyImages.attestors.entries.keys
for more details on the supported key options.
Testing
Code Block |
---|
> kubectl events -n ncyd-flux --watch
0s Warning error HelmRelease/kowl Helm install failed: 1 error occurred:
* admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Deployment/ncyd/kowl was blocked due to the following policies
check-image:
autogen-check-image: 'failed to verify image ncydacrinprogress.azurecr.io/cloudhut/kowl:v1.5.0:
.attestors[0].entries[0].keys: GET https://ncydacrinprogress.azurecr.io/oauth2/token?scope=repository%!A(MISSING)cloudhut%!F(MISSING)kowl%!A(MISSING)pull&service=ncydacrinprogress.azurecr.io:
UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization
for more information.'
Last Helm logs:
0s Warning error HelmRelease/kowl reconciliation failed: Helm install failed: 1 error occurred:
* admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Deployment/ncyd/kowl was blocked due to the following policies
check-image:
autogen-check-image: 'failed to verify image ncydacrinprogress.azurecr.io/cloudhut/kowl:v1.5.0:
.attestors[0].entries[0].keys: GET https://ncydacrinprogress.azurecr.io/oauth2/token?scope=repository%!A(MISSING)cloudhut%!F(MISSING)kowl%!A(MISSING)pull&service=ncydacrinprogress.azurecr.io:
UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization
for more information.' |
Script to Sign Helm Charts and Docker Images
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bash
#source optional env file
set -a
. ~/cosign/env
set -e +a
#************ SAMPLE ENV FILE ************
#BUILD="22.0.1-4040670"
#SRE_BUILD="23.2.0-3610795"
#OCI_REPO="xxx.azurecr.io"
#OCI_REPO_USERNAME=xxx
#OCI_REPO_PASSWORD="xxx"
#COSIGN_PRIVATE_KEY="-----BEGIN ENCRYPTED SIGSTORE PRIVATE KEY-----
#...
#-----END ENCRYPTED SIGSTORE PRIVATE KEY-----"
#COSIGN_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----
#...
#-----END PUBLIC KEY-----"
export COSIGN_PASSWORD=""
signArtifact () {
cosign sign --yes --key env://COSIGN_PRIVATE_KEY \
--registry-username="${OCI_REPO_USERNAME}" \
--registry-password="${OCI_REPO_PASSWORD}" \
${OCI_REPO}/$1
}
verifyArtifact(){
cosign verify --key env://COSIGN_PUBLIC_KEY \
--registry-username="${OCI_REPO_USERNAME}" \
--registry-password="${OCI_REPO_PASSWORD}" \
${OCI_REPO}/$1 >/dev/null
}
source images.src
source charts.src
images=(${images[@]})
charts=(${charts[@]})
requiredBins=(
"cosign"
)
for bin in "${requiredBins[@]}"; do
if ! command -v ${bin} &> /dev/null
then
echo "Required command ${bin} could not be found, please install and re-run"
exit
fi
done
echo
echo "=================="
echo "SIGNING CHARTS"
echo "=================="
for chart in "${charts[@]}"; do
chartName=`echo ${chart}|sed 's/:.*//'`
chartVersion=`echo ${chart}|sed -e 's/.*://'`
echo "ChartName: ${chartName}"
echo "ChartVersion: ${chartVersion}"
echo "------------------"
signArtifact charts/${chartName}:${chartVersion}
verifyArtifact charts/${chartName}:${chartVersion}
echo "------------------"
done
echo
echo "=================="
echo "SIGNING IMAGES"
echo "=================="
for image in "${images[@]}"; do
imageNameAndVersion=`echo $image |sed 's:[^/]*/\(.*\):\1:'`
imageName=`echo ${imageNameAndVersion}|sed -e 's/:.*//'`
imageVersion=`echo ${imageNameAndVersion}|sed -e 's/.*://'`
echo "Image Name: ${imageName}"
echo "Image Version: ${imageVersion}"
echo "------------------"
signArtifact ${imageName}:${imageVersion}
verifyArtifact ${imageName}:${imageVersion}
echo "------------------"
done
echo ""
echo "DONE!"
|
Code Block | ||
---|---|---|
| ||
images=(
pkg/fluent-bit:$BUILD"
)
|
Code Block | ||
---|---|---|
| ||
charts=(
"mychart:$BUILD"
"ckaf/kafka/rocky8:8.4.2-7.3.1-7486"
) |
Helpful Scripts
Install latest helm
Code Block |
---|
https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash |
Login to Helm Registry
Code Block |
---|
helm registry login ncydacrinprogress.azurecr.io --username ncydacrinprogress --password xxx |
Push Chart
Code Block |
---|
helm push ssh-server-0.1.0.tgz oci://ncydacrinprogress.azurecr.io/charts |
List artifacts in Azure Container Registry
Code Block |
---|
az acr login --name ncydacrinprogress.azurecr.io -u ncydacrinprogress -p xxxx
az acr repository list --name ncydacrinprogress.azurecr.io -u ncydacrinprogress -p xxxx -o tsv
|
Output
Code Block |
---|
acm/agent
acm/block-provider
acm/frontend
acm/maria-db
acm/mockserver
acm/server
atlassian/jira-software
azuremonitor/containerinsights/ciprod
bats/bats
bitnami/nginx
... |
Using ORAS
Install
Code Block |
---|
brew install oras |
Login
Code Block |
---|
oras login ${OCI_REPO} --username ${OCI_REPO_USERNAME} --password ${OCI_REPO_PASSWORD} |
List Artifacts
Code Block |
---|
oras repo ls ${OCI_REPO}/ |
Get Latest Tag
Code Block |
---|
oras repo tags ${OCI_REPO}/${artifact} | tail -n 1 |
References
Reference | URL | ||
---|---|---|---|
Cosign Quickstart | https://docs.sigstore.dev/signing/quickstart/ | ||
Prove the Authenticity of OCI Artifacts | https://fluxcd.io/blog/2022/10/prove-the-authenticity-of-oci-artifacts/ | ||
Signing and Verifying OCI Artifacts | https://fluxcd.io/flux/cheatsheets/oci-artifacts/#signing-and-verification | ||
Security: Image Provenance | https://fluxcd.io/blog/2022/02/security-image-provenance/ | ||
Sign and Verify Container Images with Docker, Cosign, and Kyverno: A Complete Guide | https://medium.com/@seifeddinerajhi/sign-and-verify-container-images-with-cosign-and-kyverno-a-complete-guide-b32b1f6e6264 | ||
Sigstore | https://kyverno.io/docs/writing-policies/verify-images/sigstore/ | 2022/10/prove-the-authenticity-of-oci-artifacts/Signing and Verifying OCI Artifacts | https://fluxcd.io/flux/cheatsheets/oci-artifacts/#signing-and-verification |