Generate certificate for registry
The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.
We can work around this for now. Not currently needed.
Create Registry Deployment and Generate htpasswd file
Define Local Storage
> vi localStorage.yml
apiVersion: v1 kind: PersistentVolume metadata: name: local-storage spec: capacity: storage: 10Gi # volumeMode field requires BlockVolume Alpha feature gate to be enabled. volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/k8s/LOCAL_STORAGE nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker1 - k8sworker2 - k8sworker3 - docker-for-desktop --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-storage-claim spec: storageClassName: local-storage accessModes: - ReadWriteOnce resources: requests: storage: 3Gi
For running on a docker for desktop cluster, you will probably need to update the path to a folder under the users home directory.
ie.
path: /Users/john.mehan/k8s/LOCAL_STORAGE
Apply the yml file
kubectl apply -f localStorage.yml
On each of the worker nodes, create the folder specified in 'path'.
ssh k8sworker1
sudo mkdir -p /var/k8s/LOCAL_STORAGE
Repeat for all worker nodes.
Create Password File
We will generate a password file to use with our registry. The default username password will be test/testpw.
vi htpasswdGenerator.yml
apiVersion: v1 kind: Pod metadata: name: htpasswd-generator spec: containers: - name: htpasswd-generator image: registry:2 command: ["/usr/bin/htpasswd"] args: ["-Bcb", "/auth/htpasswd", "test", "testpw"] volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker2 - docker-for-desktop volumes: - name: local-vol persistentVolumeClaim: claimName: local-storage-claim restartPolicy: OnFailure
kubectl apply -f htpasswdGenerator.yml
Create Registry Deployment
We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml
vi registry.yml
apiVersion: v1 kind: Pod metadata: name: registry labels: app: registry spec: containers: - name: registry image: registry:2 env: - name: REGISTRY_AUTH value: htpasswd - name: REGISTRY_AUTH_HTPASSWD_REALM value: "Registry Realm" - name: REGISTRY_AUTH_HTPASSWD_PATH value: /auth/htpasswd ports: - containerPort: 5000 volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth - mountPath: /var/lib/registry name: local-vol subPath: registry/data affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker2 - docker-for-desktop volumes: - name: local-vol persistentVolumeClaim: claimName: local-storage-claim --- apiVersion: v1 kind: Service metadata: name: registry-ext spec: type: NodePort selector: app: registry ports: - port: 5000 nodePort: 30500 name: registry-ext
kubectl apply -f registry.yml
Verify that you can access it
http://localhost:30500/v2/_catalog
or
curl --user test:testpw localhost:30500/v2/_catalog
or
docker login localhost:30500
Feed the Registry
docker tag <image> localhost:30500/<image>
docker push localhost:30500/<image>
example:
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
docker push localhost:30500/nginx-jmehan:latest
Create a secret in the cluster that holds your authorization token
> kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
$ kubectl create secret docker-registry regcred --docker-server=http://localhost:30500/ --docker-username=test --docker-password=testpw --docker-email=test@irdeto.com
Revise deployment.yaml files
Add imagePullSecrets section to your deployment yaml files:
imagePullSecrets:
- name: regcred
example:
apiVersion: v1 kind: Pod metadata: name: sample labels: app: sample spec: containers: - name: sample image: localhost:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred
Test that you can pull from the cluster registry
Create a custom docker image
mkdir html
vi html/index.html
<html> <body> <h1>Hello from John</h1> </body> </html>
vi Dockerfile
FROM nginx COPY html /usr/share/nginx/html
docker build -t nginx-jmehan .
Sending build context to Docker daemon 5.632kB Step 1/2 : FROM nginx ---> 7042885a156a Step 2/2 : COPY html /usr/share/nginx/html ---> Using cache ---> b284da22e8cb Successfully built b284da22e8cb Successfully tagged nginx-jmehan:latest
Push the Custom Image to the new Registry
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
docker push localhost:30500/nginx-jmehan:latest
Verify that it is in the registry
curl --user test:testpw localhost:30500/v2/_catalog
{"repositories":["nginx-jmehan"]}
Configure a node that uses the new image
vi sample.yml
apiVersion: v1 kind: Pod metadata: name: sample labels: app: sample spec: containers: - name: sample image: localhost:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: Service metadata: name: sample spec: type: NodePort selector: app: sample ports: - port: 80 nodePort: 30080 name: sample
kubectl apply -f sample.yml
Navigate to http://localhost:31081/ to see our new pod.
References
Reference | URL |
---|---|
Deploy a registry server | https://docs.docker.com/registry/deploying/ |
Configuring a registry | https://docs.docker.com/registry/configuration/ |
Manage TLS Certificates in a Cluster | https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#requesting-a-certificate |
Docker registry:2 Setup with TLS, Basic Auth, and persistent data | https://medium.com/@ManagedKube/docker-registry-2-setup-with-tls-basic-auth-and-persistent-data-8b98a2a73eec |
Docker Login | https://docs.docker.com/engine/reference/commandline/login/ |
Using a Private Registry | https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry |
Define a Command and Arguments for a Container | https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/ |
In Cluster Docker Registy with TLS | https://medium.com/@jmarhee/in-cluster-docker-registry-with-tls-on-kubernetes-758eecfe8254 |