Kommander GPU configuration

Configure GPU for Kommander clusters

This topic describes the following procedures for managing your Nvidia platform services:

Prerequisites

Before you begin, you must:

  • Ensure nodes provide an Nvidia GPU.

  • For AWS, select a GPU instance type from the Accelerated Computing section of the AWS instance types.

  • Run nodes on CentOS 7.

  • Perform the Node Deployment procedure.

Enable Nvidia Platform Service on Kommander

To enable Nvidia GPU support on Kommander, perform the following steps:

  1. Add the label konvoy.mesosphere.com/gpu-provider=NVIDIA to each GPU node with installed Nvidia host components.

    kubectl label node <NODE> konvoy.mesosphere.com/gpu-provider=NVIDIA

    NOTE: These placement labels will be applied automatically in later revisions.

  2. Enable the nvidia services when you install Kommander.

    helm install -n kommander --create-namespace kommander-bootstrap kommander/kommander-bootstrap --set services.nvidia.enabled=true

Disable Nvidia Platform Service on Kommander

  1. Delete all GPU workloads on the GPU nodes where the Nvidia platform service needs to be upgraded.

  2. Delete the existing Nvidia platform service.

  3. Wait for all Nvidia-related resources in the Terminating state to be cleaned up. You can check pod status with:

    kubectl get pods -A | grep nvidia

Upgrade Nvidia Platform Service on Kommander

Kommander can automatically upgrade the Nvidia GPU platform service. However, GPU workload must be drained before the Nvidia platform service can be upgraded.

To upgrade, follow the instructions to disable the service, and then the instructions to enable the service.

Nvidia GPU Monitoring

Kommander uses the NVIDIA Data Center GPU Manager to export GPU metrics towards Prometheus. By default, Kommander has a Grafana dashboard called NVIDIA DCGM Exporter Dashboard to monitor GPU metrics. This GPU dashboard is shown in Kommander’s Grafana UI.

Troubleshooting

  1. Determine if all Nvidia pods are in Running state, as expected:

    kubectl get pods -A | grep nvidia

  2. Collect the logs for any problematic Nvidia pods, if there are any crashing, returning errors, or flapping. For example:

    kubectl logs -n kube-system nvidia-nvidia-device-plugin-rpdwj

  3. To recover from this problem, you must restart all Nvidia platform service pods that are running on the SAME host. In the example below, all Nvidia resources are restarted on the node ip-10-0-101-65.us-west-2.compute.internal:

    $ kubectl get pod -A -o wide | grep nvidia
kommander                           nvidia-nvidia-dcgm-exporter-s26r7                                    1/1     Running     0          51m     192.168.167.174   ip-10-0-101-65.us-west-2.compute.internal    <none>           <none>
kommander                           nvidia-nvidia-dcgm-exporter-w7lf4                                    1/1     Running     0          51m     192.168.111.173   ip-10-0-75-212.us-west-2.compute.internal    <none>           <none>
kube-system                         nvidia-nvidia-device-plugin-rpdwj                                    1/1     Running     0          51m     192.168.167.175   ip-10-0-101-65.us-west-2.compute.internal    <none>           <none>
kube-system                         nvidia-nvidia-device-plugin-z7m2s                                    1/1     Running     0          51m     192.168.111.172   ip-10-0-75-212.us-west-2.compute.internal    <none>           <none>
$ kubectl delete pod -n kommander nvidia-nvidia-dcgm-exporter-s26r7
pod "nvidia-nvidia-dcgm-exporter-s26r7" deleted
$ kubectl delete pod -n kube-system nvidia-nvidia-device-plugin-rpdwj
pod "nvidia-nvidia-device-plugin-rpdwj" deleted

  4. To collect more debug information on the Nvidia platform service, run:

    helm get all nvidia -n kommander

  5. To validate metrics being produced and exported by the Nvidia DCGM exporter on a GPU node:

    kubectl exec -n kommander nvidia-nvidia-dcgm-exporter-s26r7 --tty -- wget -nv -O- localhost:9400/metrics