Generate certificate for registry

The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.

We can work around this for now. Not currently needed.

...

Create Registry in Cluster

Define Local Storage

> vi localStorage.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage
spec:
  capacity:
    storage: 10Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/k8s/LOCAL_STORAGE
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker1
          - k8sworker2
          - k8sworker3
          - docker-for-desktop
  
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-storage-claim
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

...

Repeat for all worker nodes.

Create Password File

We will generate a password file to use with our registry. The default username password will be test/testpw.

...

kubectl apply -f htpasswdGenerator.yml

Create SSL Certificate

Create

...

CSR

We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml

vi registry.yml

are using CFSSL to generate the Certificate Signing Request. 

Install CFSSL on your mac using the following command:

brew install cfssl

Create script to create CSR:

vi createCSR.sh

cat <<EOF | cfssl genkey - | cfssljson -bare server
{ 
  "hosts": [
    "registry-ext.default.svc.cluster.local",
    "registry.default.svc.cluster.local",
    "registry.default.pod.cluster.local"
  ],
  "CN": "registry.default.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  }
}
EOF

Run script:

./createCSR

This will generate two files: 

• server-key.pem
• server.csr

Submit CSR to Kubernetes Cluster for Approval

Login to the master node

Create script to submit CSR

vi submitCSR.sh

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: registry.default
spec:
  groups:
  - system:authenticated
  request: $(cat server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl get csr

Run the script:

./submitCSR.sh

certificatesigningrequest.certificates.k8s.io/registry.default created

Approve CSR

kubectl certificate approve registry.default

kubectl get csr

NAME       

apiVersion: v1
kind: Pod
metadata:
  name: registry
  labels:
    app: registry
spec:
  containers:
  - name: registry
    image: registry:2
    env:
    - name: REGISTRY_AUTH
      value: htpasswd
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: /auth/htpasswd
    ports:
    - containerPort: 5000
    volumeMounts:
    - mountPath: /auth
      name: local-vol
      subPath: registry/auth
    - mountPath: /var/lib/registry
      name: local-vol
      subPath: registry/data
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
        AGE   REQUESTOR operator: In
            values:
CONDITION
registry.default   30s         - k8sworker2
            - docker-for-desktop
  volumes:
  - name: local-vol
    persistentVolumeClaim:
      claimName: local-storage-claim
---
apiVersion: v1
kind: Service
metadata:
  name: registry-ext
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
kubernetes-admin   Approved,Issued

Get Signed Certificate

kubectl get csr registry.default -o jsonpath='{.status.certificate}' | base64 --decode > server.crt

Create Secret for SSL Certificate

kubectl create secret generic registry-ssl --from-file=./server.crt --from-file=./server-key.pem

kubectl get secrets

NAME                  TYPE                             nodePort: 30500
    DATA  name: registry-ext

kubectl apply -f registry.yml

Verify that you can access it

http://localhost:30500/v2/_catalog

or

curl --user test:testpw localhost:30500/v2/_catalog

or

docker login localhost:30500

Feed the Registry

docker tag <image> localhost:30500/<image>

docker push localhost:30500/<image>

example:

docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest

docker push localhost:30500/nginx-jmehan:latest

Create a secret in the cluster that holds your authorization token

> kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

...

AGE
registry-ssl          Opaque                                2      32s

Create Registry Pod and Service 

We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml

vi registry.yml

apiVersion: v1
kind: Pod
metadata:
  name: registry
  labels:
    app: registry
spec:
  containers:
  - name: registry
    image: registry:2
    env:
    - name: REGISTRY_AUTH
      value: htpasswd
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: /auth/htpasswd
    - name: REGISTRY_HTTP_ADDR
      value: 0.0.0.0:5000
    - name: REGISTRY_HTTP_TLS_CERTIFICATE
      value: /ssl/server.crt
    - name: REGISTRY_HTTP_TLS_KEY
      value: /ssl/server-key.pem
    ports:
    - containerPort: 5000
    volumeMounts:
    - mountPath: /auth
      name: local-vol
      subPath: registry/auth
    - mountPath: /var/lib/registry
      name: local-vol
      subPath: registry/data
    - mountPath: /ssl
      name: registry-ssl
      readOnly: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - k8sworker2
            - docker-for-desktop
  volumes:
  - name: local-vol
    persistentVolumeClaim:
      claimName: local-storage-claim
  - name: registry-ssl
    secret:
      secretName: registry-ssl
---
apiVersion: v1
kind: Service
metadata:
  name: registry-ext
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
      nodePort: 30500
      name: registry-ext

kubectl apply -f registry.yml

Verify that the registry was successfully installed by issuing the following command:

curl -k --user test:testpw https://<HOST_OR_IP>:30500/v2/_catalog

{"repositories":[]}

Create a secret in the Cluster for the Registry

Create a secret in the cluster that holds your authorization token

kubectl create secret docker-registry regcred --docker-server=

...

'https://

...

127.0.0.1:30500/' --docker-username='test' --docker-password='testpw' --docker-email=

...

'test@irdeto.com'

Feed the Registry

From user machine, revise the docker daemon file.

On Mac:

vi ~/.docker/daemon.json

{
  "debug" : true,
  "experimental" : false,
  "insecure-registries" : ["172.20.233.181:30500"]
}

Add an entry for insecure-registries for your cluster registry.

docker tag <image> <node_ip>:30500/<image>

docker push <node_ip>:30500/<image>

example:

docker tag nginx-jmehan:latest 172.20.233.181:30500/nginx-jmehan:latest

docker push 172.20.233.181:30500/nginx-jmehan:latest

Verify that the registry was successfully installed by issuing the following command:

curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog

{"repositories":["nginx-jmehan"]}

Revise deployment.yaml files

Add imagePullSecrets section to your deployment yaml files:

imagePullSecrets:
- name: regcred

example:

Revise deployment.yaml files

Add imagePullSecrets section to your deployment yaml files:

imagePullSecrets:
- name: regcred

example:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-jmehan
spec:
  selector:
    matchLabels:
      app: nginx-jmehan
  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx-jmehan
    spec:
      containers:
      - name: nginx-jmehan
        image: localhost:30500/nginx-jmehan:latest
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: ServicePod
metadata:
  name: sample
  labels: nginx-jmehan
    app: sample
spec:
  typecontainers: NodePort
  selector- name: sample
    appimage: 127.0.0.1:30500/nginx-jmehan:latest
    ports:
    - portcontainerPort: 80
      nodePort: 31081imagePullSecrets:
      - name: nginx-jmehan
regcred

Test that you can pull from the cluster registry

...

Sending build context to Docker daemon  5.632kB
Step 1/2 : FROM nginx
 ---> 7042885a156a
Step 2/2 : COPY html /usr/share/nginx/html
 ---> Using cacheStep 1/2 : FROM nginx
 ---> b284da22e8cb7042885a156a
SuccessfullyStep built b284da22e8cb
Successfully tagged nginx-jmehan:latest

Push the Custom Image to the new Registry

docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest

docker push localhost:30500/nginx-jmehan:latest

Verify that it is in the registry

curl --user test:testpw localhost:30500/v2/_catalog

{"repositories":["nginx-jmehan"]}

...

2/2 : COPY html /usr/share/nginx/html
 ---> Using cache
 ---> b284da22e8cb
Successfully built b284da22e8cb
Successfully tagged nginx-jmehan:latest

Push the Custom Image to the new Registry

docker tag nginx-jmehan:latest 172.20.233.181:30500/nginx-jmehan:latest

docker push 172.20.233.181:30500/nginx-jmehan:latest

Verify that it is in the registry

curl -k --user test:testpw https://172.20.233.181:30500/v2/_catalog

{"repositories":["nginx-jmehan"]}

Configure a node that uses the new image

vi sample.yml

apiVersion: v1
kind: Pod
metadata:
  name: sample
  labels:

...

vi nginx-jmehan.yml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-jmehan
spec:
  selector:
    matchLabels:
      app: nginx-jmehan
  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx-jmehan
    sample
spec:
      containers:
      - name: nginx-jmehan
    sample
    image: localhost127.0.0.1:30500/nginx-jmehan:latest
    
    ports:
        - containerPort: 80
      imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-jmehan
spec:
  type: NodePort
  selector:
    app: nginx-jmehan
  ports:
    - port: 80
      nodePort: 31081
      name: nginx-jmehan

kubectl apply -f nginx-jmehan.yml

...

metadata:
  name: sample
spec:
  type: NodePort
  selector:
    app: sample
  ports:
    - port: 80
      nodePort: 30080
      name: sample

kubectl apply -f sample.yml

Navigate to http://localhost:30080/ to see our new pod.

Troubleshooting

Start a ubuntu pod

kubectl run -it --image=ubuntu:latest bash

From bash shell of new pod:

apt-get update

apt-get install curl

curl -k --user test:testpw https://registry:5000/v2/_catalog

{"repositories":["nginx-jmehan"]}

References

...