Prerequisites
Before performing the uninstall, ensure you have:
kubectl
installed, and configured to connect to your cluster.jq
installed, and available in your Linux PATH.
Uninstall Kaptain
Uninstalling Kaptain requires the execution of several manual steps.
-
Delete all profiles configured with Kaptain. Profiles have dependencies (such as finalizers) associated with Kaptain, so they must be removed before uninstalling Kaptain. Attempting to uninstall Kaptain without removing the profiles will fail and leave many resources in a broken state.
The following command deletes all notebooks, pipelines and deployed models permanently. Notebook volumes created when deploying notebooks are cleaned up and deleted.
kubectl delete profiles.kubeflow.org --all
DO NOT proceed until all profiles are removed. If a profile cannot be removed, review the troubleshooting section for instructions to [forcibly delete a profile][#cannot-finalize-removing-a-profile].
-
Uninstall Kaptain using KUDO. This command deletes the Kaptain Percona DB store, all deployments, statefulsets, jobs, and volumes associated with the kubeflow namespace.
kubectl kudo uninstall --instance "kaptain" --namespace kubeflow --wait --wait-time=600
-
Ensure that the
kubeflow
namespace is removed:kubectl delete namespace kubeflow
-
Clean up any additional configuration resources left behind by KUDO-managed Kaptain, so you can install Helm-managed Kaptain correctly. This command selects various resource types with the label
kudo.dev/instance
of either"kaptain"
or beginning with"kaptain-"
.for type in clusterrole customresourcedefinition clusterrolebinding clusterrole mutatingwebhookconfiguration ValidatingWebhookConfiguration operators; do INSTANCES=$(kubectl get $type --output json | jq '.items[].metadata.labels["kudo.dev/instance"] | select(. != null)' -r | egrep '^kaptain(-|$)' | tr "\n" ",") kubectl delete $type --selector "kudo.dev/instance in ($INSTANCES)" done
Kaptain is now fully uninstalled.
-
Remove Kudo:
kubectl kudo init --upgrade --dry-run --output yaml | kubectl delete -f -
Troubleshooting and overcoming failures
Sometimes, the uninstall steps will fail or hang because a cleanup task gets stuck. In that case, you will need to force Kubernetes to uninstall the resources. In such cases, you may need to clean up things manually. Follow the steps below as necessary.
Cannot finalize removing a profile
-
Skip the finalizer and manually clean up the namespace and related persistent volumes.
kubectl patch profile <profile-name> -p '{"metadata":{"finalizers":null}}' --type=merge
-
Try to delete the profile again:
kubectl delete profile <profile-name>
-
Find, and make a record of all persistent volumes IDs associated with this profile. You will delete these volumes later.
kubectl get persistentvolume
-
Delete the namespace associated with the profile.
kubectl delete namespace <profile-name>
-
Delete any persistent volumes that were associated with this profile.
kubectl delete persistentvolume <persistent-volume-id>
KUDO Kaptain uninstall fails
If KUDO Kaptain uninstall hangs and fails to properly uninstall, force the removal of the Kaptain instance, and manually clean up volumes and namespaces.
-
Find the KUDO instance which is stuck:
kubectl get instances -n kubeflow
-
Patch it, and remove it:
kubectl patch instances -n kubeflow <instance-name> -p '{"metadata":{"finalizers":null}}' --type=merge kubectl delete instance <instance-name>
-
Repeat the above until
kubectl get instances -n kubeflow
returns an empty list. -
Find, and make a record of all persistent volumes IDs associated with Kubeflow. Look at the CLAIM column and look for values such as
kubeflow/datadir-kaptain-mysql-store-pxc-0
. You will delete these volumes later.kubectl get persistentvolume
-
Remove the kubeflow namespace:
kubectl delete namespace kubeflow
-
Remove all dangling persistent volumes:
kubectl delete persistentvolume <persistentvolumeid>