Generate certificate for registry

The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.

We can work around this for now. Not currently needed.

...

Create Registry in Cluster

Define Local Storage

> vi localStorage.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage
spec:
  capacity:
    storage: 10Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/k8s/LOCAL_STORAGE
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker1
          - k8sworker2
          - k8sworker3
          - docker-for-desktop
  
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-storage-claim
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

...

Repeat for all worker nodes.

Create Password File

We will generate a password file to use with our registry. The default username password will be test/testpw.

...

kubectl apply -f htpasswdGenerator.yml

...

Create SSL Certificate

Create CSR

We are using CFSSL to generate the Certificate Signing Request. 

Install CFSSL on your mac using the following command:

brew install cfssl

Create script to create CSR:

vi createCSR.sh

cat <<EOF | cfssl genkey - | cfssljson -bare server
{ 
  "hosts": [
    "registry-ext.default.svc.cluster.local",
    "registry.default.svc.cluster.local",
    "registry.default.pod.cluster.local"
  ],
  "CN": "registry.default.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  }
}
EOF

Run script:

./createCSR

This will generate two files: 

• server-key.pem
• server.csr

Submit CSR to Kubernetes Cluster for Approval

Login to the master node

Create script to submit CSR

vi submitCSR.sh

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: registry.default
spec:
  groups:
  - system:authenticated
  request: $(cat server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl get csr

Run the script:

./submitCSR.sh

certificatesigningrequest.certificates.k8s.io/registry.default created

Approve CSR

kubectl certificate approve registry.default

kubectl get csr

NAME               AGE   REQUESTOR          CONDITION
registry.default   30s   kubernetes-admin   Approved,Issued

Get Signed Certificate

kubectl get csr registry.default -o jsonpath='{.status.certificate}' | base64 --decode > server.crt

Create Secret for SSL Certificate

kubectl create secret generic registry-ssl --from-file=./server.crt --from-file=./server-key.pem

kubectl get secrets

NAME                  TYPE                                  DATA   AGE
registry-ssl          Opaque                                2      32s

Create Registry Pod and Service 

We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml

...

apiVersion: v1
kind: Pod
metadata:
  name: registry
  labels:
    app: registry
spec:
  containers:
  - name: registry
    image: registry:2
    env:
    - name: REGISTRY_AUTH
      value: htpasswd
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: /auth/htpasswd
    - name: REGISTRY_HTTP_ADDR
      value: 0.0.0.0:5000
    - name: REGISTRY_HTTP_TLS_CERTIFICATE
      value: /ssl/server.crt
    - name: REGISTRY_HTTP_TLS_KEY
      value: /ssl/server-key.pem
    ports:
    - containerPort: 5000
    volumeMounts:
    - mountPath: /auth
      name: local-vol
      subPath: registry/auth
    - mountPath: /var/lib/registry
      name: local-vol
      subPath: registry/data
    - mountPath: /ssl
      name: registry-ssl
      readOnly: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - k8sworker2
            - docker-for-desktop
  volumes:
  - name: local-vol
    persistentVolumeClaim:
      claimName: local-storage-claim
  - name: registry-ssl
    secret:
      secretName: registry-ssl
---
apiVersion: v1
kind: Service
metadata:
  name: registry-ext
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
      nodePort: 30500
      name: registry-ext

kubectl apply -f registry.yml

Verify that you can access it

http://localhost:30500/v2/_catalog

or

...

the registry was successfully installed by issuing the following command:

curl -k --user test:testpw

...

https://<HOST_OR_IP>:30500/v2/_catalog

or

docker login localhost:30500

Feed the Registry

docker tag <image> localhost:30500/<image>

docker push localhost:30500/<image>

example:

docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest

{"repositories":[]}

Create a secret in the Cluster for the Registry

...

Create a secret in the cluster that holds your authorization token>

kubectl create secret docker-registry regcred --docker-server=

...

'https://127.0.0.1:30500/' --docker-username=

...

'test' --docker-password=

...

'testpw' -

...

-

...

docker-email=

...

'test@irdeto.com'

Feed the Registry

From user machine, revise the docker daemon file.

On Mac:

vi ~/.docker/daemon.json

{
  "debug" : true,
  "experimental" : false,
  "insecure-registries" : ["172.20.233.181:30500"]
}

Add an entry for insecure-registries for your cluster registry.

docker tag <image> <node_ip>:30500/<image>

docker push <node_ip>:30500/<image>

example:

docker tag nginx-jmehan:latest 172.20.233.181:30500/nginx-jmehan:latest

docker push 172.20.233.181:30500/nginx-jmehan:latest

Verify that the registry was successfully installed by issuing the following command:

curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog

{"repositories":["nginx-jmehan"]}

Revise deployment.yaml files

...

apiVersion: v1
kind: Pod
metadata:
  name: sample
  labels:
    app: sample
spec:
  containers:
  - name: sample
    image: localhost127.0.0.1:30500/nginx-jmehan:latest
    ports:
    - containerPort: 80
  imagePullSecrets:
      - name: regcred

...

docker tag nginx-jmehan:latest localhost 172.20.233.181:30500/nginx-jmehan:latest

docker push localhost172.20.233.181:30500/nginx-jmehan:latest

Verify that it is in the registry

curl -k --user test:testpw localhost https://172.20.233.181:30500/v2/_catalog

{"repositories":["nginx-jmehan"]}

Configure a node that uses the new image

vi sample.yml

apiVersion: v1
kind: Pod
metadata:
  name: sample
  labels:
    app: sample
spec:
  containers:
  - name: sample
    image: localhost127.0.0.1:30500/nginx-jmehan:latest
    ports:
    - containerPort: 80
  imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: sample
spec:
  type: NodePort
  selector:
    app: sample
  ports:
    - port: 80
      nodePort: 30080
      name: sample

...

Navigate to http://localhost:30080/ to see our new pod.

Troubleshooting

Start a ubuntu pod

kubectl run -it --image=ubuntu:latest bash

From bash shell of new pod:

apt-get update

apt-get install curl

curl -k --user test:testpw https://registry:5000/v2/_catalog

{"repositories":["nginx-jmehan"]}

References

...