Minikube Setup
https://minikube.sigs.k8s.io/docs/tutorials/using_psp/
Install minikube on Mac with Apple Silicon
| Code Block |
|---|
|
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-arm64 /usr/local/bin/minikube |
Start Minikube with Kubernetes version 1.21.5 running Enabling Pod Security Policies in Docker for Desktop.
| Code Block |
|---|
|
dockerminikube run -itstart --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy --privilegedaddons=pod-security-policy --piddriver=hostdocker debian nsenter -t 1 -malsologtostderr -u -n -i sh |
| Code Block |
|---|
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
172730635f67: Pull complete
Digest: sha256:e538a2f0566efc44db21503277c7312a142f4d0dedc5d2886932b92626104bff
Status: Downloaded newer image for debian:latest
/ #
/ #
/ # vi /etc/kubernetes/manifests/kube-apiserver.yaml |
kubernetes-version=v1.21.5 |
| Code Block |
|---|
...
I1014 10:25:53.474818 47355 out.go:177] Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default |
Apply PSPs
By default, minikube will apply the following pod security policies, cluster roles and bindings:
| Code Block |
|---|
|
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- "*"
volumes:
- "*"
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
|
| Code Block |
|---|
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.65.
creationTimestamp: null
labels: max: 65535
fsGroup:
rule: 'MustRunAs'
componentranges:
kube-apiserver # Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:privileged
labels:
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
tierverbs: control-plane ['use']
resourceNames:
- privileged
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- restricted
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: default:restricted
labels:
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-apiserver psp:restricted
subjects:
- kind: Group
name: system:authenticated
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: default:privileged
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:privileged
subjects:
- kind: Group
name: system:masters
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:kube-system
apiGroup: rbac.authorization.k8s.io
|
Test
| Code Block |
|---|
|
vi dangerous-pod.yaml |
| Code Block |
|---|
|
apiVersion: apps/v1
kind: Deployment
metadata:
name: dangerous-deploy
labels:
app: dangerous-deploy
spec:
replicas: 1
selector:
matchLabels:
app: dangerous-deploy
template:
metadata:
labels:
app: dangerous-deploy
spec:
containers: containers:
- commandname: alpine
image: alpine
stdin: true
tty: true
securityContext:
privileged: true
hostPID: true
- kube-apiserver
- --advertise-address=192.168.65.4
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/run/config/pki/ca.crt
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy |
Restart Docker for Desktop.
https://minikube.sigs.k8s.io/docs/tutorials/using_psp/
Sample PodSecurityPolicy - privileged
| Code Block |
|---|
|
kubectl apply -f dangerous-pod.yaml |
You should notice that it doesn't deploy. Now test some other installations and see what needs to be done to bypass.
Fluent-bit Example
For fluent-bit, we are going to define a separate pod security policy and bind it to fluent-bit's service account.
| Code Block |
|---|
|
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: ncyd-fluent-bit-psp
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'hostPath'
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup |
| Code Block |
|---|
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
kubernetes.io/description: 'privileged allows full unrestricted access to
pod features, as if the PodSecurityPolicy controller was not enabled.'
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
labels:
kubernetes.io/cluster-service: "true"
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAnyMustRunAs'
seLinux ranges:
# Forbid adding the rule: 'RunAsAny'
supplementalGroups:
root group.
- rulemin: 'RunAsAny'1
fsGroup:
rulemax: 'RunAsAny'65535
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: podsecuritypolicy:privilegedncyd-fluent-bit
labels:
addonmanager.kubernetes.io/cluster-servicemode: "true" EnsureExists
rules:
- apiGroups:
- policy['policy']
resourceNames:resources: ['podsecuritypolicies']
-verbs: privileged
resources:
- podsecuritypolicies['use']
verbsresourceNames:
- use
ncyd-fluent-bit-psp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: podsecuritypolicy:authenticated
annotationsncyd-fluent-bit
subjects:
kubernetes.io/description: 'Allow all authenticated users to create privileged pods.'- kind: ServiceAccount
labelsname: fluent-bit
kubernetes.io/cluster-servicenamespace: "true" default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: podsecuritypolicy:privileged
subjects:
- kind: Groupncyd-fluent-bit
apiGroup: rbac.authorization.k8s.io
|
Apply:
| Code Block |
|---|
kubectl apply name: system:authenticated-f <psp_file> |
References