Table of Contents |
---|
Generate certificate for registry
The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.
We can work around this for now. Not currently needed.
...
Create Registry in Cluster
Define Local Storage
> vi localStorage.yml
Code Block | ||
---|---|---|
| ||
apiVersion: v1 kind: PersistentVolume metadata: name: local-storage spec: capacity: storage: 10Gi # volumeMode field requires BlockVolume Alpha feature gate to be enabled. volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/k8s/LOCAL_STORAGE nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - k8sworker1 - k8sworker2 - k8sworker3 - docker-for-desktop --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: local-storage-claim spec: storageClassName: local-storage accessModes: - ReadWriteOnce resources: requests: storage: 3Gi |
...
Repeat for all worker nodes.
Create Password File
We will generate a password file to use with our registry. The default username password will be test/testpw.
...
kubectl apply -f htpasswdGenerator.yml
Create SSL Certificate
Create
...
CSR
We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml
vi registry.yml
are using CFSSL to generate the Certificate Signing Request.
Install CFSSL on your mac using the following command:
brew install cfssl
Create script to create CSR:
vi createCSR.sh
Code Block | ||||
---|---|---|---|---|
| ||||
cat <<EOF | cfssl genkey - | cfssljson -bare server
{
"hosts": [
"registry-ext.default.svc.cluster.local",
"registry.default.svc.cluster.local",
"registry.default.pod.cluster.local"
],
"CN": "registry.default.pod.cluster.local",
"key": {
"algo": "ecdsa",
"size": 256
}
}
EOF |
Run script:
./createCSR
This will generate two files:
- server-key.pem
- server.csr
Submit CSR to Kubernetes Cluster for Approval
Login to the master node
Create script to submit CSR
vi submitCSR.sh
Code Block | ||||
---|---|---|---|---|
| ||||
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: registry.default
spec:
groups:
- system:authenticated
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
kubectl get csr |
Run the script:
./submitCSR.sh
Code Block |
---|
certificatesigningrequest.certificates.k8s.io/registry.default created |
Approve CSR
kubectl certificate approve registry.default
kubectl get csr
Code Block | ||
---|---|---|
NAME | ||
Code Block | ||
| ||
apiVersion: v1 kind: Pod metadata: name: registry labels: app: registry spec: containers: - name: registry image: registry:2 env: - name: REGISTRY_AUTH value: htpasswd - name: REGISTRY_AUTH_HTPASSWD_REALM value: "Registry Realm" - name: REGISTRY_AUTH_HTPASSWD_PATH value: /auth/htpasswd ports: - containerPort: 5000 volumeMounts: - mountPath: /auth name: local-vol subPath: registry/auth - mountPath: /var/lib/registry name: local-vol subPath: registry/data affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: -AGE key: kubernetes.io/hostname REQUESTOR operator: In CONDITION registry.default 30s values: - k8sworker2 - docker-for-desktop volumes: - name: local-vol persistentVolumeClaim: kubernetes-admin Approved,Issued |
Get Signed Certificate
kubectl get csr registry.default -o jsonpath='{.status.certificate}' | base64 --decode > server.crt
Create Secret for SSL Certificate
kubectl create secret generic registry-ssl --from-file=./server.crt --from-file=./server-key.pem
kubectl get secrets
Code Block |
---|
NAME claimName: local-storage-claim --- apiVersion: v1 kind: Service metadata: name: registry-ext spec: type: NodePort selector: TYPE app: registry ports: - port: 5000 nodePort: 30500 DATA name: registry-ext |
kubectl apply -f registry.yml
Verify that you can access it
http://localhost:30500/v2/_catalog
or
curl --user test:testpw localhost:30500/v2/_catalog
or
docker login localhost:30500
Feed the Registry
docker tag <image> localhost:30500/<image>
docker push localhost:30500/<image>
example:
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
docker push localhost:30500/nginx-jmehan:latest
Create a secret in the cluster that holds your authorization token
> kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
...
AGE
registry-ssl Opaque 2 32s |
Create Registry Pod and Service
We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml
vi registry.yml
Code Block | ||
---|---|---|
| ||
apiVersion: v1
kind: Pod
metadata:
name: registry
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2
env:
- name: REGISTRY_AUTH
value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: "Registry Realm"
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /auth/htpasswd
- name: REGISTRY_HTTP_ADDR
value: 0.0.0.0:5000
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: /ssl/server.crt
- name: REGISTRY_HTTP_TLS_KEY
value: /ssl/server-key.pem
ports:
- containerPort: 5000
volumeMounts:
- mountPath: /auth
name: local-vol
subPath: registry/auth
- mountPath: /var/lib/registry
name: local-vol
subPath: registry/data
- mountPath: /ssl
name: registry-ssl
readOnly: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8sworker2
- docker-for-desktop
volumes:
- name: local-vol
persistentVolumeClaim:
claimName: local-storage-claim
- name: registry-ssl
secret:
secretName: registry-ssl
---
apiVersion: v1
kind: Service
metadata:
name: registry-ext
spec:
type: NodePort
selector:
app: registry
ports:
- port: 5000
nodePort: 30500
name: registry-ext
|
kubectl apply -f registry.yml
Verify that the registry was successfully installed by issuing the following command:
curl -k --user test:testpw https://<HOST_OR_IP>:30500/v2/_catalog
Code Block |
---|
{"repositories":[]} |
Create a secret in the Cluster for the Registry
Create a secret in the cluster that holds your authorization token
kubectl create secret docker-registry regcred --docker-server=
...
...
//127.0.0.1:30500/' --docker-username='test' --docker-password='testpw' --docker-email=
...
'test@irdeto.com
...
'
Feed the Registry
From user machine, revise the docker daemon file.
On Mac:
vi ~/.docker/daemon.json
Code Block |
---|
{
"debug" : true,
"experimental" : false,
"insecure-registries" : ["172.20.233.181:30500"]
} |
Add an entry for insecure-registries for your cluster registry.
docker tag <image> <node_ip>:30500/<image>
docker push <node_ip>:30500/<image>
example:
docker tag nginx-jmehan:latest 172.20.233.181:30500/nginx-jmehan:latest
docker push 172.20.233.181:30500/nginx-jmehan:latest
Verify that the registry was successfully installed by issuing the following command:
curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog
Code Block |
---|
{"repositories":["nginx-jmehan"]} |
Revise deployment.yaml files
Add imagePullSecrets section to your deployment yaml files:
imagePullSecrets:
- name: regcred
example:
Code Block | ||
---|---|---|
|
Revise deployment.yaml files
Add imagePullSecrets section to your deployment yaml files:
imagePullSecrets:
- name: regcred
example:
Code Block | ||
---|---|---|
| ||
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-jmehan spec: selector: matchLabels: app: nginx-jmehan replicas: 1 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx-jmehan spec: containers: - name: nginx-jmehan image: localhost:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: ServicePod metadata: name: sample labels: app: nginx-jmehansample spec: typecontainers: NodePort - selectorname: sample appimage: 127.0.0.1:30500/nginx-jmehan:latest ports: - port: 80 containerPort: 80 nodePortimagePullSecrets: 31081 - name: nginx-jmehan regcred |
Test that you can pull from the cluster registry
...
docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
Configure a node that uses the new image
vi nginx-jmehan.yml
172.20.233.181:30500/nginx-jmehan:latest
docker push 172.20.233.181:30500/nginx-jmehan:latest
Verify that it is in the registry
curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog
Code Block |
---|
{"repositories":["nginx-jmehan"]} |
Configure a node that uses the new image
vi sample.yml
Code Block | ||
---|---|---|
| ||
apiVersion: v1
kind: Pod
metadata:
name: sample
labels:
| ||
Code Block | ||
| ||
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-jmehan spec: selector: matchLabels: app: nginx-jmehan replicas: 1 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx-jmehan sample spec: containers: - name: nginx-jmehan sample image: localhost127.0.0.1:30500/nginx-jmehan:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- apiVersion: v1 kind: Service metadata: name: sample spec: type: NodePort selector: app: sample ports: - port: 80 name: nginx-jmehan spec: typenodePort: NodePort30080 selector: app: nginx-jmehan ports: - port: 80 nodePort: 31081 name: nginx-jmehan |
kubectl apply -f nginx-jmehan.yml
...
name: sample
|
kubectl apply -f sample.yml
Navigate to http://localhost:30080/ to see our new pod.
Troubleshooting
Start a ubuntu pod
kubectl run -it --image=ubuntu:latest bash
From bash shell of new pod:
apt-get update
apt-get install curl
curl -k --user test:testpw https://registry:5000/v2/_catalog
{"repositories":["nginx-jmehan"]}
References
Reference | URL |
---|---|
Deploy a registry server | https://docs.docker.com/registry/deploying/ |
Configuring a registry | https://docs.docker.com/registry/configuration/ |
Manage TLS Certificates in a Cluster | https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#requesting-a-certificate |
Docker registry:2 Setup with TLS, Basic Auth, and persistent data | https://medium.com/@ManagedKube/docker-registry-2-setup-with-tls-basic-auth-and-persistent-data-8b98a2a73eec |
Docker Login | https://docs.docker.com/engine/reference/commandline/login/ |
Using a Private Registry | https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry |
Define a Command and Arguments for a Container | https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/ |
In Cluster Docker Registy with TLS | https://medium.com/@jmarhee/in-cluster-docker-registry-with-tls-on-kubernetes-758eecfe8254 |
...