1091 ibm cpd scheduler not deleted by cp4d delete instancesh

[luigimolinaro]

Bug Fixes

Fixed Scheduler Namespace Deletion

• Issue: Script used hardcoded ibm-scheduling namespace instead of the configurable variable
• Fix: Now uses PROJECT_SCHEDULING_SERVICE variable (default: cpd-scheduler)

Removed Duplicate Deletion Attempts

• Eliminated redundant namespace deletion calls in delete_ibm_license_server() and delete_ibm_certificate_manager()
• Improved script efficiency and reduced unnecessary API calls

New Features

1. Force Finalizer Removal (--force-finalizer)

• Enables forced removal of Kubernetes finalizers via OpenShift REST API
• Helps resolve stuck namespaces in Terminating state
• Works on both namespace-level and resource-level finalizers

2. Configurable Timeout (--timeout <SECONDS>)

• Set custom timeout for namespace deletion (default: 900 seconds)
• Prevents indefinite waiting on stuck operations
• Example: --timeout 1200 for 20-minute timeout

3. Automatic Retry Logic

• Up to 3 automatic retry attempts when timeout is reached
• Shorter timeout (300s) for retry attempts
• Automatic forced cleanup on retry if --force-finalizer is enabled

4. Parallel Namespace Deletion (--parallel)

• Delete multiple namespaces simultaneously for significantly faster execution
• Up to 3x faster than sequential deletion
• Reduces total deletion time from ~15-30 minutes to ~5-10 minutes
• Maintains safety by keeping instance and operator namespace deletions sequential
• Parallel deletion applies to cluster-wide namespaces only

5. Colored Output

• ✓ Green (log_success): Successful operations
• ✗ Red (log_error): Errors and failures
• ⚠ Yellow (log_warning): Warnings and timeouts
• ℹ Cyan (log_info): Informational messages
• Makes it easy to quickly identify operation status

Robust Cleanup Functions

force_remove_resource_finalizers()

Removes finalizers from resources that commonly block namespace deletion:

• PersistentVolumeClaims (PVCs): Often have finalizers preventing deletion
• PersistentVolumes (PVs): Associated volumes that may be stuck
• Pods in Terminating state: Force deleted with --grace-period=0
• Services: Remove finalizers blocking service cleanup
• ConfigMaps & Secrets: Clean up resources with finalizers

diagnose_namespace_stuck()

Provides comprehensive diagnostic information when namespaces are stuck:

• Lists all remaining resources in the namespace
• Shows namespace status and finalizers
• Identifies pods in Terminating state
• Lists PVCs that may be blocking deletion
• Helps troubleshoot deletion issues

Enhanced wait_ns_deleted()

• Progress logging every 60 seconds
• Automatic diagnostics on timeout
• Retry logic with forced cleanup
• Better error handling and return codes

[fketelaars]

• Enables forced removal of Kubernetes finalizers via OpenShift REST API --> Why use REST API whereas in other parts of the code, the oc patch command is used?
• When trying the command with a non-existing CP4D instance namespace, the command tries to delete ' ' namespace and waits 900 seconds
• Parallel should be the default

[fketelaars]

This is what I mean:

cp4d-delete-instance.sh cpd --parallel
About to delete the following from the cluster:
- Instance namespace:
- Operator namespace:
- IBM Custom Resource Definitions
Are you sure (y/N)? y
ℹ [2026-03-13 07:12:21] Using parallel deletion mode for faster execution
error: resource(s) were provided, but no name was specified
[2026-03-13 07:12:21] Getting Custom Resources in OpenShift project ...
You must specify the type of resource to get. Use "oc api-resources" for a complete list of supported resources.

error: Required resource not specified.
Use "oc explain <resource>" for a detailed description of that resource (e.g. oc explain pods).
See 'oc get -h' for help and examples
[2026-03-13 07:12:22] Delete all Custom Resources except the base ones
[2026-03-13 07:12:22] Delete remaining Custom Resources
[2026-03-13 07:12:22] Delete role binding if Cloud Pak for Data was connected to IAM
error: resource(s) were provided, but no name was specified
error: the server doesn't have a resource type "authentication"
[2026-03-13 07:12:22] Waiting for deletion of namespace  (timeout: 900s)...

[luigimolinaro]

This is what I mean:

cp4d-delete-instance.sh cpd --parallel
About to delete the following from the cluster:
- Instance namespace:
- Operator namespace:
- IBM Custom Resource Definitions
Are you sure (y/N)? y
ℹ [2026-03-13 07:12:21] Using parallel deletion mode for faster execution
error: resource(s) were provided, but no name was specified
[2026-03-13 07:12:21] Getting Custom Resources in OpenShift project ...
You must specify the type of resource to get. Use "oc api-resources" for a complete list of supported resources.

error: Required resource not specified.
Use "oc explain <resource>" for a detailed description of that resource (e.g. oc explain pods).
See 'oc get -h' for help and examples
[2026-03-13 07:12:22] Delete all Custom Resources except the base ones
[2026-03-13 07:12:22] Delete remaining Custom Resources
[2026-03-13 07:12:22] Delete role binding if Cloud Pak for Data was connected to IAM
error: resource(s) were provided, but no name was specified
error: the server doesn't have a resource type "authentication"
[2026-03-13 07:12:22] Waiting for deletion of namespace  (timeout: 900s)...

My assumption is that the environment variables normally used to manage Cloud Pak for Data are set.

For example, variables like:

export PROJECT_CERT_MANAGER="cert-manager"
export PROJECT_LICENSE_SERVICE="ibm-licensing"
export PROJECT_SCHEDULING_SERVICE="cpd-scheduler"
export PROJECT_CPD_INST_OPERATORS="cpd-operators"
export PROJECT_CPD_INST_OPERANDS="cpd"

If these variables are not defined, the script has no way to determine the namespaces and resources it should operate on.

This variables are usually very clear when Cloud Pak is installed without the deployer, because these variables are explicitly configured. However, when using the Cloud Pak Deployer, users often rely only on the deployer abstraction and may not be aware of these underlying variables.

I'm currently thinking about how we could improve the script so it can automatically discover these values instead of relying on environment variables.

Let me think about the best way to implement this.
If you have any suggestions on how we could detect the namespaces or resources dynamically, they would be very welcome. 👍

[luigimolinaro]

I think i find a way :

=== Discovered Cloud Pak for Data Configuration ===
Instance namespace:   cpd
Operator namespace:   cpd-operators
Scheduler namespace:  <not found>
Cert-manager:         cert-manager
Licensing:            ibm-licensing
==================================================

Please check

[fketelaars]

I'm not sure if the changes are heading in the right direction. Concerns:

• cert-manager and the operator namespace are Red Hat. I don't want this script to touch these, only the IBM certificate manager if it exists.
• I really don't want auto-discovery for the CP4D instance namespace; this is too risky IMO. Let users specify the namespace and assume that the operator namespace is <instance_namespace>-operators or it must be specified at command line or environment variable
• There are situations where instance namespace was deleted (or pending deletion) and the operator namespace was not, also vice-versa. Don't limit the script by assuming that these are still there; I want it to clean up residuals as well, even if orphaned.