Pod Priority and Preemption

Pod Priority is used to apply importance of a pod relative to other pods. In this section we will create two PriorityClasses and watch the interaction of pods.

Create PriorityClass

We will create two PriorityClass, low-priority and high-priority.

cat <<EoF > ~/environment/resource-management/high-priority-class.yml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 100
globalDefault: false
description: "High-priority Pods"
EoF

kubectl apply -f ~/environment/resource-management/high-priority-class.yml


cat <<EoF > ~/environment/resource-management/low-priority-class.yml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 50
globalDefault: false
description: "Low-priority Pods"
EoF

kubectl apply -f ~/environment/resource-management/low-priority-class.yml

Pods with without a PriorityClass are 0. A global PriorityClass can be assigned. Additional details can be found here

Deploy low-priority Pods

Next we will deploy low-priority pods to use up resources on the nodes. The goal is to saturate the nodes with as many pods as possible.

cat <<EoF > ~/environment/resource-management/low-priority-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-deployment
  name: nginx-deployment
spec:
  replicas: 50
  selector:
    matchLabels:
      app: nginx-deployment
  template:
    metadata:
      labels:
        app: nginx-deployment
    spec:
      priorityClassName: "low-priority"      
      containers:            
       - image: nginx
         name: nginx-deployment
         resources:
           limits:
              memory: 1G  
EoF
kubectl apply -f ~/environment/resource-management/low-priority-deployment.yml

Watch the number of available pods in the Deployment until the available stabilizes around a number. This exercise does not require all pods in the deployment to be in Available state. We want to ensure the nodes are completely filled with pods. It may take up to 2 minutes to stabilize.

kubectl get deployment nginx-deployment --watch

Output: 


NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   5/50    50           5           5s
nginx-deployment   6/50    50           6           6s
...
nginx-deployment   21/50   50           21          20s
nginx-deployment   21/50   50           21          6m

Deploy High Priority Pod

In a new terminal watch Deployment using the command below

kubectl get deployment --watch

Next deploy high-priority Deployment to see the how Kubernetes handles PriorityClass.

cat <<EoF > ~/environment/resource-management/high-priority-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: high-nginx-deployment
  name: high-nginx-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: high-nginx-deployment
  template:
    metadata:
      labels:
        app: high-nginx-deployment
    spec:
      priorityClassName: "high-priority"      
      containers:            
       - image: nginx
         name: high-nginx-deployment
         resources:
           limits:
              memory: 1G
EoF
kubectl apply -f ~/environment/resource-management/high-priority-deployment.yml

What changes did you see?

Expand for output