Generate certificate for registry
The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.
We can work around this for now. Not currently needed.
Create Registry Deployment and Generate htpasswd file
Define Local Storage
> vi localStorage.yml
apiVersion: v1 kind: PersistentVolume metadata: name: local-storage spec: capacity: storage: 10Gi # volumeMode field requires BlockVolume Alpha feature gate to be enabled. volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/k8s/LOCAL_STORAGE nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker1 - k8sworker2 - k8sworker3 - docker-for-desktop --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-storage-claim spec: storageClassName: local-storage accessModes: - ReadWriteOnce resources: requests: storage: 3Gi
ssh k8sworker1
sudo mkdir -p /var/k8s/LOCAL_STORAGE
Repeat for all nodes requiring local storage.
From master node
kubectl apply -f localStorage.yml
Create Password File
We will generate a password file to use with our registry. The default username password will be test/testpw.
vi htpasswdGenerator.yml
apiVersion: v1 kind: Pod metadata: name: htpasswd-generator spec: containers: - name: htpasswd-generator image: registry:2 command: ["/usr/bin/htpasswd"] args: ["-Bcb", "/auth/htpasswd", "test", "testpw"] volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth volumes: - name: local-vol persistentVolumeClaim: claimName: local-storage-claim restartPolicy: OnFailure
kubectl apply -f htpasswdGenerator.yml
Create Registry Deployment
We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml
vi registry.yml
apiVersion: v1 kind: Pod metadata: name: registry labels: app: registry spec: containers: - name: registry image: registry:2 env: - name: REGISTRY_AUTH value: htpasswd - name: REGISTRY_AUTH_HTPASSWD_REALM value: "Registry Realm" - name: REGISTRY_AUTH_HTPASSWD_PATH value: /auth/htpasswd ports: - containerPort: 5000 volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth - mountPath: /var/lib/registry name: local-vol subPath: registry/data nodeSelector: kubernetes.io/hostname: k8sworker2 volumes: - name: local-vol persistentVolumeClaim: claimName: local-storage-claim --- apiVersion: v1 kind: Service metadata: name: registry-ext spec: type: NodePort selector: app: registry ports: - port: 5000 nodePort: 30500 name: registry-ext
kubectl apply -f registry.yml
Verify that you can access it
http://localhost:30500/v2/_catalog
or
curl --user test:testpw localhost:30500/v2/_catalog
or
docker login localhost:30500
Feed the Registry
docker tag <image> localhost:30500/<image>
docker push localhost:30500/<image>
example:
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
docker push localhost:30500/nginx-jmehan:latest
Create a secret in the cluster that holds your authorization token
> kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
$ kubectl create secret docker-registry regcred --docker-server=http://localhost:30500/ --docker-username=test --docker-password=testpw --docker-email=test@irdeto.com
Revise deployment.yaml files
Add imagePullSecrets section to your deployment yaml files:
imagePullSecrets:
- name: regcred
example:
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-jmehan spec: selector: matchLabels: app: nginx-jmehan replicas: 1 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx-jmehan spec: containers: - name: nginx-jmehan image: localhost:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: Service metadata: name: nginx-jmehan spec: type: NodePort selector: app: nginx-jmehan ports: - port: 80 nodePort: 31081 name: nginx-jmehan
Test that you can pull from the cluster registry
Create a custom docker image
mkdir html
vi html/index.html
<html> <body> <h1>Hello from John</h1> </body> </html>
vi Dockerfile
FROM nginx COPY html /usr/share/nginx/html
docker build -t nginx-jmehan .
Sending build context to Docker daemon 5.632kB Step 1/2 : FROM nginx ---> 7042885a156a Step 2/2 : COPY html /usr/share/nginx/html ---> Using cache ---> b284da22e8cb Successfully built b284da22e8cb Successfully tagged nginx-jmehan:latest
Push the Custom Image to the new Registry
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
Configure a node that uses the new image
vi nginx-jmehan.yml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-jmehan spec: selector: matchLabels: app: nginx-jmehan replicas: 1 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx-jmehan spec: containers: - name: nginx-jmehan image: localhost:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: Service metadata: name: nginx-jmehan spec: type: NodePort selector: app: nginx-jmehan ports: - port: 80 nodePort: 31081 name: nginx-jmehan
kubectl apply -f nginx-jmehan.yml
Navigate to http://localhost:31081/ to see our new pod.
References
Reference | URL |
---|---|
Deploy a registry server | https://docs.docker.com/registry/deploying/ |
Configuring a registry | https://docs.docker.com/registry/configuration/ |
Manage TLS Certificates in a Cluster | https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#requesting-a-certificate |
Docker registry:2 Setup with TLS, Basic Auth, and persistent data | https://medium.com/@ManagedKube/docker-registry-2-setup-with-tls-basic-auth-and-persistent-data-8b98a2a73eec |
Docker Login | https://docs.docker.com/engine/reference/commandline/login/ |
Using a Private Registry | https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry |
Define a Command and Arguments for a Container | https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/ |
In Cluster Docker Registy with TLS | https://medium.com/@jmarhee/in-cluster-docker-registry-with-tls-on-kubernetes-758eecfe8254 |