Table of Contents |
---|
Generate certificate for registry
The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.
We can work around this for now. Not currently needed.
...
Create Registry in Cluster
Define Local Storage
> vi localStorage.yml
Code Block | ||
---|---|---|
| ||
apiVersion: v1 kind: PersistentVolume metadata: name: local-storage spec: capacity: storage: 10Gi # volumeMode field requires BlockVolume Alpha feature gate to be enabled. volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/k8s/LOCAL_STORAGE nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker1 - k8sworker2 - k8sworker3 - docker-for-desktop --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-storage-claim spec: storageClassName: local-storage accessModes: - ReadWriteOnce resources: requests: storage: 3Gi |
...
Repeat for all worker nodes.
Create Password File
We will generate a password file to use with our registry. The default username password will be test/testpw.
...
kubectl apply -f htpasswdGenerator.yml
...
Create SSL Certificate
Create CSR
We are using CFSSL to generate the Certificate Signing Request.
Install CFSSL on your mac using the following command:
brew install cfssl
Create script to create CSR:
vi createCSR.sh
Code Block | ||||
---|---|---|---|---|
| ||||
cat <<EOF | cfssl genkey - | cfssljson -bare server
{
"hosts": [
"registry-ext.default.svc.cluster.local",
"registry.default.svc.cluster.local",
"registry.default.pod.cluster.local"
],
"CN": "registry.default.pod.cluster.local",
"key": {
"algo": "ecdsa",
"size": 256
}
}
EOF |
Run script:
./createCSR
This will generate two files:
- server-key.pem
- server.csr
Submit CSR to Kubernetes Cluster for Approval
Login to the master node
Create script to submit CSR
vi submitCSR.sh
Code Block | ||||
---|---|---|---|---|
| ||||
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: registry.default
spec:
groups:
- system:authenticated
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
kubectl get csr |
Run the script:
./submitCSR.sh
Code Block |
---|
certificatesigningrequest.certificates.k8s.io/registry.default created |
Approve CSR
kubectl certificate approve registry.default
kubectl get csr
Code Block |
---|
NAME AGE REQUESTOR CONDITION
registry.default 30s kubernetes-admin Approved,Issued |
Get Signed Certificate
kubectl get csr registry.default -o jsonpath='{.status.certificate}' | base64 --decode > server.crt
Create Secret for SSL Certificate
kubectl create secret generic registry-ssl --from-file=./server.crt --from-file=./server-key.pem
kubectl get secrets
Code Block |
---|
NAME TYPE DATA AGE
registry-ssl Opaque 2 32s |
Create Registry Pod and Service
We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml
...
Code Block | ||
---|---|---|
| ||
apiVersion: v1 kind: Pod metadata: name: registry labels: app: registry spec: containers: - name: registry image: registry:2 env: - name: REGISTRY_AUTH value: htpasswd - name: REGISTRY_AUTH_HTPASSWD_REALM value: "Registry Realm" - name: REGISTRY_AUTH_HTPASSWD_PATH value: /auth/htpasswd - name: REGISTRY_HTTP_ADDR value: 0.0.0.0:5000 - name: REGISTRY_HTTP_TLS_CERTIFICATE value: /ssl/server.crt - name: REGISTRY_HTTP_TLS_KEY value: /ssl/server-key.pem ports: - containerPort: 5000 volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth - mountPath: /var/lib/registry name: local-vol subPath: registry/data - mountPath: /ssl name: registry-ssl readOnly: true affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker2 - docker-for-desktop volumes: - name: local-vol persistentVolumeClaim: claimName: local-storage-claim - name: registry-ssl secret: secretName: registry-ssl --- apiVersion: v1 kind: Service metadata: name: registry-ext spec: type: NodePort selector: app: registry ports: - port: 5000 nodePort: 30500 name: registry-ext |
kubectl apply -f registry.yml
Verify that you can access it
http://localhost:30500/v2/_catalog
or
...
the registry was successfully installed by issuing the following command:
curl -k --user test:testpw
...
https://<HOST_OR_IP>:30500/v2/_catalog
or
docker login localhost:30500
Feed the Registry
docker tag <image> localhost:30500/<image>
docker push localhost:30500/<image>
example:
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
Code Block |
---|
{"repositories":[]} |
Create a secret in the Cluster for the Registry
...
Create a secret in the cluster that holds your authorization token>
kubectl create secret docker-registry regcred --docker-server=
...
'https://127.0.0.1:30500/' --docker-username=
...
'test' -
...
-
...
docker-password=
...
'testpw' --docker-email=
...
'test@irdeto.com'
Feed the Registry
From user machine, revise the docker daemon file.
On Mac:
vi ~/.docker/daemon.json
Code Block |
---|
{
"debug" : true,
"experimental" : false,
"insecure-registries" : ["172.20.233.181:30500"]
} |
Add an entry for insecure-registries for your cluster registry.
docker tag <image> <node_ip>:30500/<image>
docker push <node_ip>:30500/<image>
example:
docker tag nginx-jmehan:latest 172.20.233.181:30500/nginx-jmehan:latest
docker push 172.20.233.181:30500/nginx-jmehan:latest
Verify that the registry was successfully installed by issuing the following command:
curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog
Code Block |
---|
{"repositories":["nginx-jmehan"]} |
Revise deployment.yaml files
...
Code Block | ||
---|---|---|
| ||
apiVersion: v1 kind: Pod metadata: name: sample labels: app: sample spec: containers: - name: sample image: localhost127.0.0.1:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred |
...
docker tag nginx-jmehan:latest localhost 172.20.233.181:30500/nginx-jmehan:latest
docker push localhost172.20.233.181:30500/nginx-jmehan:latest
Verify that it is in the registry
curl -k --user test:testpw localhost https://172.20.233.181:30500/v2/_catalog
Code Block |
---|
{"repositories":["nginx-jmehan"]} |
Configure a node that uses the new image
vi sample.yml
Code Block | ||
---|---|---|
| ||
apiVersion: v1 kind: Pod metadata: name: sample labels: app: sample spec: containers: - name: sample image: localhost127.0.0.1:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: Service metadata: name: sample spec: type: NodePort selector: app: sample ports: - port: 80 nodePort: 30080 name: sample |
kubectl apply -f sample.yml
Navigate to http://localhost:3108130080/ to see our new pod.
Troubleshooting
Start a ubuntu pod
kubectl run -it --image=ubuntu:latest bash
From bash shell of new pod:
apt-get update
apt-get install curl
curl -k --user test:testpw https://registry:5000/v2/_catalog
{"repositories":["nginx-jmehan"]}
References
Reference | URL |
---|---|
Deploy a registry server | https://docs.docker.com/registry/deploying/ |
Configuring a registry | https://docs.docker.com/registry/configuration/ |
Manage TLS Certificates in a Cluster | https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#requesting-a-certificate |
Docker registry:2 Setup with TLS, Basic Auth, and persistent data | https://medium.com/@ManagedKube/docker-registry-2-setup-with-tls-basic-auth-and-persistent-data-8b98a2a73eec |
Docker Login | https://docs.docker.com/engine/reference/commandline/login/ |
Using a Private Registry | https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry |
Define a Command and Arguments for a Container | https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/ |
In Cluster Docker Registy with TLS | https://medium.com/@jmarhee/in-cluster-docker-registry-with-tls-on-kubernetes-758eecfe8254 |
...