How to upgrade Kubernetes cluster with kubeadm

How to upgrade Kubernetes cluster with kubeadm

Overview:

Testbed scheme:

kuber2.png

The overall steps to upgrade Kubernetes cluster are:

  • Upgrade all nodes from v1.23.0 --> v1.24.1 To see the latest kubeadm releases go to https://kubernetes.io/releases/
  • Upgrade Kubernetes control plane
  • Upgrade worker nodes

Kubernetes control plane upgrade (control node):

  1. To upgrade the control node first we need to drain the node:
    kubectl drain [node_name] --ignore-daemonsets
    
  2. Upgrade the kubeadm (v1.23.0 --> v1.24.1):
    sudo apt-get update && \
    sudo apt-get install -y --allow-change-held-packages kubeadm=1.24.1-00
    
  3. Check the kubeadm version:
    kubeadm version
    
  4. To update the Kubernetes control plane internal components use:
    sudo kubeadm upgrade plan v1.24.1
    
    You should see the following message:
  5. Now apply the upgrade:
    sudo kubeadm upgrade apply v1.24.1
    
    You should see the following message:
  6. Upgrade kubelet and kubectl packages:
    sudo apt-get update && \
    sudo apt-get install -y --allow-change-held-packages kubelet=1.24.1-00 kubectl=1.24.1-00
    
  7. Reload the daemon:
    sudo systemctl daemon-reload
    
  8. Restart the kubelet:
    sudo systemctl restart kubelet
    
  9. After an upgrade made you can now uncordon the node and back the workload to the node again:
    kubectl uncordon [control_node_name]
    
  10. Check the upgrade with the following command:
    kubectl get nodes
    
    You now should see that control node is upgraded to version 1.24.1:

Worker nodes upgrade :

  1. Now drain the worker node (control node):
    kubectl drain [worker_node_name] --ignore-daemonsets --force
    
    ATTENTION: --force flag is dangerous as it will evict all standalone pods not controlled by Kubernetes controller objects
  2. Now log into your drained worker node and upgrade the kubeadm:
    sudo apt-get update && \
    sudo apt-get install -y --allow-change-held-packages kubeadm=1.24.1-00
    
  3. Now upgrade the node:
    sudo kubeadm upgrade node
    
  4. Upgrade the kubelet and kubectl packages:
    sudo apt-get update && \
    sudo apt-get install -y --allow-change-held-packages kubelet=1.24.1-00 kubectl=1.24.1-00
    
  5. Reload the daemon:
    sudo systemctl daemon-reload
    
  6. Restart the kubelet:
    sudo systemctl restart kubelet
    
  7. Now go back to the control node and uncordon the worker node:
    kubectl uncordon [worker_node_name]
    
  8. Check if worker node is upgraded:
    kubectl get nodes
    
    You should see the following:
  9. Repeat the same process for the rest of worker nodes

Notes:

  1. Node draining is the mechanism that allows users to gracefully move all containers from one node to the other ones.
  2. A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the node that a Pod runs on is selected by the Kubernetes scheduler. However, DaemonSet pods are created and scheduled by the DaemonSet controller instead.
  3. --allow-change-held-packages - overwrites the hold status from kubeadm package. Hold status prevents any upgrades of the package
  4. systemctl is a Linux command-line utility used to control and manage systemd and services. You can think of Systemctl as a control interface for Systemd init service, allowing you to communicate with systemd and perform operations. Systemctl is a successor of Init.

Reference:

  1. Safely Drain a Node
  2. How Daemon Pods are scheduled
  3. What is systemctl?