Registry Inside a Cluster

Generate certificate for registry

The Kubernetes cluster has a CA Authority and can create certificates that can be consumed by pods in the cluster.

We can work around this for now. Not currently needed.

Create Registry Deployment and Generate htpasswd file

Define Local Storage

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage
spec:
  capacity:
    storage: 10Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/k8s/LOCAL_STORAGE
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker1
          - k8sworker2
          - k8sworker3
          - docker-for-desktop
  
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-storage-claim
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

ssh k8sworker1
sudo mkdir -p /var/k8s/LOCAL_STORAGE

Repeat for all nodes requiring local storage.

From master node

kubectl apply -f localstorage.yml

Create Password File

We will generate a password file to use with our registry. The default username password will be test/testpw.

vi htpasswd-generator.yml

apiVersion: v1
kind: Pod
metadata:
  name: htpasswd-generator
spec:
  containers:
  - name: htpasswd-generator
    image: registry:2
    command: ["/usr/bin/htpasswd"]
    args: ["-Bcb", "/auth/htpasswd", "test", "testpw"]
    volumeMounts:
        - mountPath: /auth
          name: local-vol
          subPath: registry/auth
  volumes:
  - name: local-vol
    persistentVolumeClaim:
      claimName: local-storage-claim
  restartPolicy: OnFailure

kubectl apply -f htpasswd-generator

Create Registry Deployment

We will create our registry and tie it to node k8sworker2. See nodeSelector in following registry.yml

vi registry.yml

registry.yml

apiVersion: v1
kind: Pod
metadata:
  name: registry
  labels:
    app: registry
spec:
  containers:
  - name: registry
    image: registry:2
    env:
    - name: REGISTRY_AUTH
      value: htpasswd
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: /auth/htpasswd
    ports:
    - containerPort: 5000
    volumeMounts:
    - mountPath: /auth
      name: local-vol
      subPath: registry/auth
    - mountPath: /var/lib/registry
      name: local-vol
      subPath: registry/data
  nodeSelector:
    kubernetes.io/hostname: k8sworker2
  volumes:
  - name: local-vol
    persistentVolumeClaim:
      claimName: local-storage-claim
---
apiVersion: v1
kind: Service
metadata:
  name: registry-ext
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
      nodePort: 30500
      name: registry-ext

kubectl apply -f registry.yml

Verify that you can access it

http://localhost:30500/v2/_catalog

or

curl --user test:testpw localhost:30500/v2/_catalog

or

docker login localhost:30500

Feed the Registry

docker tag <image> localhost:30500/<image>

docker push localhost:30500/<image>

example:

docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest
docker push localhost:30500/nginx-jmehan:latest

Create a secret in the cluster that holds your authorization token

> kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

$ kubectl create secret docker-registry regcred --docker-server=http://localhost:30500/ --docker-username=test --docker-password=testpw --docker-email=t est@irdeto.com

Revise deployment.yaml files

Add imagePullSecrets section to your deployment yaml files:

imagePullSecrets:
- name: regcred

example:

Example: nginx pulled from registry

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-jmehan
spec:
  selector:
    matchLabels:
      app: nginx-jmehan
  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx-jmehan
    spec:
      containers:
      - name: nginx-jmehan
        image: localhost:30500/nginx-jmehan:latest
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-jmehan
spec:
  type: NodePort
  selector:
    app: nginx-jmehan
  ports:
    - port: 80
      nodePort: 31081
      name: nginx-jmehan

Test that you can pull from the cluster registry

Create a custom docker image

mkdir html
vi html/index.html

<html>
<body>
<h1>Hello from John</h1>
</body>
</html>

vi Dockerfile

FROM nginx
COPY html /usr/share/nginx/html

docker build -t nginx-jmehan .

Sending build context to Docker daemon  5.632kB
Step 1/2 : FROM nginx
 ---> 7042885a156a
Step 2/2 : COPY html /usr/share/nginx/html
 ---> Using cache
 ---> b284da22e8cb
Successfully built b284da22e8cb
Successfully tagged nginx-jmehan:latest

Push the Custom Image to the new Registry

docker tag nginx-jmehan:latest localhost:30500/nginx-jmehan:latest

Configure a node that uses the new image

vi nginx-jmehan.yml