Upgrade Kubernetes and Addons with Konvoy

Upgrade the Kubernetes version and platform service addons

Before upgrading, keep in mind there is inherent risk in upgrading any Kubernetes cluster. Any failure or error can result in unexpected downtime or loss of data. Take necessary precautions before starting the upgrade process. For example, back up the cluster state and all cluster-related files using Velero before you upgrade.

Also keep in mind that cluster addons require a specific minimum version of Kubernetes to be installed. You can verify your installed version before upgrading by running the following command:

kubectl version --short=true

A Kubernetes version upgrade using Konvoy CLI consists of a few distinct steps.

  • Update the cluster.yaml file with the changes outlined below.
  • Run konvoy up --upgrade which first upgrades the version of Kubernetes on all of the control-plane nodes. The command then upgrades the rest of the nodes, the platform service addons, and installs any additional addons specified in the cluster.yaml file.

Supported Kubernetes version upgrades

Upgrading Kubernetes is supported when upgrading to a newer release in the current or next minor version. Upgrading to a newer minor release of Kubernetes, requires a new Konvoy minor release. Upgrades that skip one or more minor versions are not supported. Downgrades are not supported. Kubernetes and Konvoy follow semantic versioning that follows the logic MAJOR.MINOR.PATCH.

For example, Konvoy supports upgrading Kubernetes v1.19.x directly to a newer v1.19.x (Konvoy v1.7.x) or Kubernetes v1.20.x (Konvoy v1.8.x). Upgrading a Kubernetes v1.18 release directly to a v1.20 release is unsupported, but Kubernetes v1.18 can be upgraded to v1.19 and then to v1.20.

This support policy applies to every node in the cluster. When upgrading a cluster running a mix of Kubernetes versions, each node upgrade must be supported for the cluster upgrade to be supported.

If you attempt an unsupported upgrade, Konvoy exits with an error. To force an unsupported upgrade, use the --force-upgrade flag.

Before you begin

Before upgrading your Kubernetes cluster, or any of the addons, you must download the specific version of Konvoy. Follow the steps outlined on the download page. You should also confirm the version of Konvoy is compatible with the version of Kubernetes you are upgrading to.

You must also have access to the cluster.yaml file and the SSH keys that were generated during the initial install. If you are using one of the public cloud provisioners you must also have access to the state/ directory that was generated during the initial install.

Prepare for Kubernetes upgrade

In the cluster.yaml file, modify the spec.kubernetes.version value in ClusterConfiguration to the desired Kubernetes version.

For example, assume the cluster was launched with the following ClusterConfiguration section:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  kubernetes:
    version: 1.18.14

If you want to upgrade to a newer minor version of Kubernetes v1.xx.x, change the version string like the following:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  kubernetes:
    version: 1.19.7

Prepare for addons upgrade

Konvoy platform service addons are managed by a library that pulls default configuration details from the kubernetes-base-addons repository.

Versioning for the platform service addons is managed by git tags and github releases within the kubernetes-base-addons repository.

Addons are deployed to the cluster as part of the konvoy up, konvoy deploy or konvoy deploy addons commands. These commands use the version of kubernetes-base-addons declared in the cluster.yaml configuration file using the spec.addons.configVersion setting.

For example, assume the cluster was launched with the following ClusterConfiguration section:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  addons:
  - configRepository: https://github.com/mesosphere/kubernetes-base-addons
    configVersion: stable-1.16-1
    addonsList:
    ...

If you want to upgrade to a newer version of addons, you must then change the version string like the following:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta2
spec:
  addons:
  - configRepository: https://github.com/mesosphere/kubernetes-base-addons
    configVersion: stable-1.17-1.2.0
    addonsList:
    ...

NOTE: The configVersion string value is of the form: stable-a.b-x.y.z.

NOTE: Depending on the addon version you are upgrading to, you may need to include additional addons. For the full list of addons refer to the reference document.

During the addons upgrade process, Konvoy upgrades the platform service addons or installs any additional addons specified in the cluster.yaml file. Also confirm the version of addons you are upgrading to is compatible with your version of Konvoy.

Upgrade the cluster

After you have modified the cluster.yaml file, you can start the upgrade process. You can upgrade all nodes, or choose to upgrade specific node pools.

During the Kubernetes upgrade process, Konvoy:

  • Determines which nodes do not have the required configuration and OS package versions.
  • Determines if upgrading these nodes from their current state to the provided cluster configuration is supported.
    • To force an unsupported upgrade, use the --force-upgrade flag.
  • During stage STAGE [Determining Upgrade Safety ...], checks for any user workloads that can be impacted by the upgrade and marks the nodes, where the workloads are running, as “unsafe” to upgrade, skipping the upgrade process on them.
    • To ignore the safety check for worker nodes only, use the --force-upgrade flag.
    • Otherwise you can resolve the safety issues after the initial upgrade and rerun the upgrade process to let Konvoy perform the upgrade on the remaining nodes.
  • Upgrades all the control-plane nodes serially.
  • Upgrades the remaining nodes sequentially, upgrading all of the nodes in a node pool before continuing onto a different node pool.
    • By default, Konvoy upgrades every node pool. To upgrade a subset of node pools, specify a comma-separated list of node pool names, with the --target-node-pools flag.
    • By default, 15% of all nodes in each node pool (except those in the control-plane) are upgraded in parallel. To change this behavior, pass the flag --max-parallel-nodes when running konvoy up, konvoy deploy or konvoy deploy kubernetes. The value can be an integer, representing the number of nodes, or a percentage of nodes in the node pool (e.g. --max-parallel-nodes 20%). Passing a value of 1 upgrades the nodes serially. If some nodes in a batch fail to upgrade, Konvoy continues to upgrade the other nodes in the batch, but exits with an error. Fix the error manually and retry the process.
    • The workloads are moved to a different node with kubectl drain. This process takes a few minutes before all workloads are scheduled onto different nodes. You can disable this behavior by passing the flag --without-draining when running konvoy up, konvoy deploy or konvoy deploy kubernetes. When --without-draining is specified, because no workloads are rescheduled, all nodes are upgraded, even when there are workloads that are “unsafe” to upgrade.

      WARNING: Using the --without-draining flag is volatile and can result in undefined behavior, downtime, and outages for your services during an upgrade. For production systems, notify your end users of cluster maintenance prior to upgrading with this flag.

    • The Kubernetes and Containerd OS packages are upgraded.
    • The node is uncordoned allowing for workloads to be scheduled on it again.

Upgrade all nodes

Start the upgrade process by running the following Konvoy command:

konvoy up --upgrade -y
This process will take about 15 minutes to complete (additional time may be required for larger clusters)

...

STAGE [Determining Upgrade Safety For Control Plane Nodes]
...

STAGE [Upgrading Kubernetes]
...

STAGE [Determining Upgrade Safety For Nodes In Pool "worker"]
...

STAGE [Upgrading Kubernetes]
...

STAGE [Determining Upgrade Safety For Nodes]
...

STAGE [Upgrading Kubernetes]
...

STAGE [Deploying Enabled Addons]
...

You can also separate the above process by first running konvoy deploy kubernetes --upgrade and then running konvoy deploy addons.

Upgrade specific node pools

You can have more control over the upgrade process by upgrading specific node pools. For example, you can upgrade only your control plane node pool, or only the control plane node pool and a single worker node pool.

NOTE: Node pools allow the cluster administrator to use different configurations for different sets of worker nodes in a heterogeneous environment. For more, see the Node Pools page.

For example, if your cluster has three worker node pools, red, yellow, and green, you can upgrade the red and yellow worker node pools. To start the upgrade process, perform the following Konvoy command:

konvoy up --upgrade -y --target-node-pools red,yellow

NOTE: Kubernetes requires that the control plane version be greater than or equal to the version of any worker node, so the control plane node pool must be upgraded before any worker node pools.

If you choose to upgrade a worker node pool, and the control plane node pool has not yet been upgraded, Konvoy automatically upgrades the control plane node pool first.

You can upgrade all remaining nodes by running the following Konvoy command:

konvoy up --upgrade -y

NOTE: If you change your cluster.yaml file to also include an upgraded version of Konvoy, using the command konvoy up --upgrade will upgrade the Kubernetes version and the CLI version of Konvoy. If you wish to upgrade both the Konvoy CLI version and Kubernetes version, follow the steps detailed in the upgrade CLI instructions.