#1 Global Leader in Data Protection & Ransomware Recovery

Resolving Pending State During Cluster Removal with k10multicluster Tool

KB ID: 4598
Product: Veeam Kasten for Kubernetes
Published: 2024-06-11
Last Modified: 2024-06-11
mailbox
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please, try again later.

Challenge

This article helps in finding and deleting long-running finalizers that are waiting on other resources and can cause deletions to appear stuck in the terminating state.

Description:

When Kubernetes attempts to delete the cluster object that has finalizers “dist.kio.kasten.io/debootstrap or dist.kio.kasten.io/manual-debootstrap” specified, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202-status code (HTTP "Accepted"). The cluster object remains in a terminating/Pending state while the control plane, or other components, take the actions defined by the finalizers

Long-running finalizers that are waiting on other resources can cause deletions to appear stuck in the terminating state. This is a result of the ServiceAccount (or its corresponding token/bindings) being deleted on the secondary cluster, and 401 or 500 error shown in the Veeam Kasten for Kubernetes dashboard.

Observation:

The Veeam Kasten for Kubernetes multi-cluster dashboard displays ‘Removal Pending’ for primary and secondary clusters, as shown below:

401
500

Solution

The traditional way to resolve this involves determining which API service is not responding and then debugging that issue.

An alternative workaround for this issue is to remove the clusters by manually deleting the finalizers which are restricting deletion of the cluster. 

Here are the steps to remove an object’s finalizers:

  1. List down the target cluster.
kubectl get clusters.dist.kio.kasten.io -n kasten-io-mc
  1. Delete the target cluster resources by its name.

kubectl delete clusters.dist.kio.kasten.io ${CLUSTER_NAME} -n kasten-io-mc
  1. Get the finalizer name to be deleted.

kubectl get cluster.dist.kio.kasten.io -n kasten-io -o yaml

Example output:

— apiVersion: dist.kio.kasten.io/v1alpha1
kind: Cluster
metadata:
 creationTimestamp: "2022-04-26T11:02:22Z"
 deletionGracePeriodseconds: 0
 deletionTimestamp: "2022-04-26T11:18:54Z"
 finalizers:
 — dist.kio.kasten.io/manual—debootstrap
 generation: 2
 labels:
   dist.kio.kasten.io/cluster—type: secondary
 name: secondary—cluster
 namespace: kasten—io—mc
 resourceVersion: "3630125"
 uid: 8eae79ca-0a3e-454e-938e-7ac3c281ba4c
  1. Manually remove an object’s finalizers by patching the metadata.finalizers field to null:

kubectl patch --namespace=kasten-io-mc clusters.dist.kio.kasten.io ${CLUSTER_NAME} -p '{"metadata":{"finalizers":null}}' --type=merge type=merge 

More Information

The following command can be used to delete all clusters at once:
for x in $(kubectl get clusters.dist.kio.kasten.io -n kasten-io-mc | awk 'NR>1 {print $1}') ; do kubectl patch --namespace=kasten-io-mc clusters.dist.kio.kasten.io $x -p '{"metadata":{"finalizers":null}}' --type=merge ; done 
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please, try again later.

You have selected too large block!

Please try select less.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please, try again later.