...
Start Minikube with Kubernetes version 1.21.5 running in Docker for Desktop.
| Code Block |
|---|
|
minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy --addons=pod-security-policy --driver=docker --alsologtostderr --kubernetes-version=v1.21.5 |
Basic Setup of privileged and restricted PSPs
| Code Block |
|---|
...
I1014 10:25:53.474818 47355 out.go:177] Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default |
Apply PSPs
By default, minikube will apply the following pod security policies, cluster roles and bindings:
| Code Block |
|---|
|
| Code Block |
|---|
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- "*"
volumes:
- "*"
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:privileged
labels:
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- privileged
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- restricted
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: default:restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted
subjects:
- kind: Group
name: system:authenticated
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: default:privileged
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:privileged
subjects:
- kind: Group
name: system:masters:
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:kube-system
apiGroup: rbac.authorization.k8s.io |
Docker for Desktop Setup - Doesn't Work
Enabling Pod Security Policies in Docker for Desktop
Test
| Code Block |
|---|
|
dockervi run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh |
| Code Block |
|---|
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
172730635f67: Pull complete
Digest: sha256:e538a2f0566efc44db21503277c7312a142f4d0dedc5d2886932b92626104bff
Status: Downloaded newer image for debian:latest
/ #
/ #
/ # vi /etc/kubernetes/manifests/kube-apiserver.yaml |
| Code Block |
|---|
|
apiVersion: apps/v1
kind: Deployment
metadata:
name: dangerous-deploy
labels:
app: dangerous-deploy
spec:
replicas: 1
selector:
matchLabels:
|
| Code Block |
|---|
apiVersion: v1 app: dangerous-deploy
template:
metadata:
labels:
app: dangerous-deploy
spec:
containers:
- name: alpine
image: alpine
kind: Pod stdin: true
tty: true
securityContext:
privileged: true
hostPID: true
hostNetwork: true
|
| Code Block |
|---|
|
kubectl apply -f dangerous-pod.yaml |
You should notice that it doesn't deploy. Now test some other installations and see what needs to be done to bypass.
Fluent-bit Example
For fluent-bit, we are going to define a separate pod security policy and bind it to fluent-bit's service account.
| Code Block |
|---|
|
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: ncyd-fluent-bit-psp
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
labels:
kubeadmaddonmanager.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.65.mode: EnsureExists
spec:
creationTimestampprivileged: nullfalse
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'hostPath'
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
labelshostPID: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
componentranges:
kube-apiserver # Forbid adding the root group.
- min: 1
max: 65535
tierreadOnlyRootFilesystem: control-plane false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-apiserverncyd-fluent-bit
labels:
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.65.4
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/run/config/pki/ca.crt
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy |
...
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- ncyd-fluent-bit-psp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ncyd-fluent-bit
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: default
roleRef:
kind: ClusterRole
name: ncyd-fluent-bit
apiGroup: rbac.authorization.k8s.io
|
Apply:
| Code Block |
|---|
kubectl apply -f <psp_file> |
References