Add cleanup for dpf scoped service interfaces

crates/api-core/src/cfg/README.md

@@ -718,7 +718,7 @@ events, so consumers handle them identically.
 | Field | Type | Default | Description |
 | ------- | ------ | --------- | ------------- |
 | `enabled` | `bool` | `false` | Enable DPF Kubernetes deployment. |
-| `deployment_scoped_service_interfaces` | `bool` | `false` | Opt the complete DPF namespace into deployment-scoped `-bf3`, `-bf4`, and `-astra` DPUServiceInterfaces. Each resource selects Nodes in the remote DPU cluster through DPF's propagated `svc.dpu.nvidia.com/owned-by-dpudeployment=<namespace>_<deployment_name>` ownership label; management-cluster DPUNode deployment labels are not used for this selector. Enabling or disabling is a planned migration: stop NICo, remove old-mode NICo ServiceInterfaces in both transition directions, perform DPU re-ingestion, and restart. NICo neither detects nor deletes old-mode resources; skipping cleanup can leave competing interface generations active. Astra requires this setting. |
+| `deployment_scoped_service_interfaces` | `bool` | `false` | Legacy migration knob for deployment-scoped DPUServiceInterfaces. BF3-only and BF4-generic-only sites remain unscoped by default for backward compatibility. BF4 Astra requires this setting so the whole DPF namespace uses scoped ServiceInterfaces and legacy match-all ServiceInterfaces cannot also bind Astra nodes. NICo removes ServiceInterfaces from the opposite mode during a transition. A failed transition fails NICo DPF initialization and is reported in its startup/reconciliation logs. It is not rolled back: resolve the reported deletion or apply error and retry initialization; either generation may be partially present. |
 | `pf_total_sf_reserved` | `u32` | `30` | SF capacity reserved beyond the NICo-managed HBN, DHCP, and FMDS endpoints when an intercept-bridging inventory is configured. NICo sets `PF_TOTAL_SF` to the effective inventory's endpoint count plus this value for BF3 and generic BF4. Without configured intercept bridging, this value is the complete `PF_TOTAL_SF`, preserving the legacy default of `30`; BF4 Astra retains its fixed flavor and ignores this setting. Changing this value changes the BF3/generic-BF4 flavor. Every intercept-inventory change requires controlled ServiceInterface cleanup and DPU re-ingestion, even when the serialized flavor and its hash remain unchanged. Operators must select a value compatible with their platform's SF and BAR capacity. With configured intercept bridging, startup rejects configurations whose managed endpoint count plus reserve exceeds `u32::MAX`. |
 | `dpu_service_sync_enabled` | `bool` | `true` | Whether NICo rolls a changed DPUService out on its own, by releasing the DPF maintenance hold on hosts whose DPUs already match their DPUDeployment. Selects *who* opens the gate, never whether one exists: DPF is always configured to park a changed DPUService behind a hold, so no service update reaches a DPU unchecked. Setting `false` does not resume unchecked rollout — the held DPUs wait for an operator to release them deliberately. Hosts still awaiting reprovisioning, and hosts carrying a live tenant instance, keep their hold either way. |
 | `dpu_agent_bootstrap_ca` | `DpfDpuAgentBootstrapCa` | `legacy_download` | Bootstrap trust for the containerized DPU agent. Supports `legacy_download` and `mounted`, as described in the following examples. |

crates/api-core/src/cfg/file.rs

@@ -1569,8 +1569,10 @@ pub struct DpfConfig {
     #[serde(default)]
     pub enabled: bool,
     /// Opts the DPF namespace into deployment-scoped DPUServiceInterfaces.
-    /// Changing modes requires operators to remove old-mode NICo resources and
-    /// re-ingest DPUs; NICo neither detects nor deletes those resources.
+    /// BF4 Astra requires this to be enabled for the whole namespace so legacy
+    /// match-all ServiceInterfaces do not bind Astra nodes. Initialization
+    /// removes ServiceInterfaces from the opposite mode during a transition;
+    /// failures are not rolled back and require a retry after remediation.
     #[serde(default)]
     pub deployment_scoped_service_interfaces: bool,
     /// SF capacity reserved beyond configured NICo-managed service endpoints.

crates/dpf/src/repository/kube.rs

@@ -556,6 +556,16 @@ impl DpuServiceInterfaceRepository for KubeRepository {
             )
             .await?)
     }
+
+    async fn delete(&self, name: &str, namespace: &str) -> Result<(), DpfError> {
+        let api: Api<DPUServiceInterface> = self.api(namespace);
+        match api.delete(name, &Default::default()).await {
+            Ok(_) => {}
+            Err(kube::Error::Api(error)) if error.code == 404 => {}
+            Err(error) => return Err(error.into()),
+        }
+        Ok(())
+    }
 }
 
 #[async_trait]

crates/dpf/src/repository/traits.rs

@@ -235,6 +235,7 @@ pub trait DpuServiceInterfaceRepository: Send + Sync {
     ) -> Result<Option<DPUServiceInterface>, DpfError>;
     async fn list(&self, namespace: &str) -> Result<Vec<DPUServiceInterface>, DpfError>;
     async fn apply(&self, iface: &DPUServiceInterface) -> Result<DPUServiceInterface, DpfError>;
+    async fn delete(&self, name: &str, namespace: &str) -> Result<(), DpfError>;
 }
 
 /// Repository for Kubernetes ConfigMaps and Secrets.

crates/dpf/src/sdk.rs

@@ -107,6 +107,8 @@ const MAX_HBN_SERVICE_INTERFACES: usize = 32;
 /// Label set by DPF on deployment-owned resources and propagated to the corresponding
 /// DPU-cluster Node. Value format: `<namespace>_<deployment_name>`.
 const DPU_OWNED_BY_DEPLOYMENT_LABEL: &str = "svc.dpu.nvidia.com/owned-by-dpudeployment";
+const SERVICE_INTERFACE_DELETE_TIMEOUT: Duration = Duration::from_secs(60);
+const SERVICE_INTERFACE_DELETE_POLL_INTERVAL: Duration = Duration::from_millis(250);
 
 /// Returns DPF's canonical ownership-label value for one DPUDeployment.
 fn dpu_deployment_owner_label_value(namespace: &str, deployment_name: &str) -> String {
@@ -664,8 +666,8 @@ pub fn deployment_cr_suffix(deployment_type: DpuDeploymentType) -> &'static str
 /// Suffix appended to deployment-scoped DPUServiceInterface CR names.
 ///
 /// Unlike the existing service CR compatibility scheme, every deployment type
-/// is suffixed. The migration is opt-in; operators must remove the old
-/// generation manually because NICo neither detects nor deletes it.
+/// is suffixed. Scoped initialization prunes the old unscoped generation so
+/// match-all legacy resources do not bind the same DPU nodes.
 fn service_interface_cr_suffix(deployment_type: DpuDeploymentType) -> &'static str {
     match deployment_type {
         DpuDeploymentType::Bf3 => "bf3",
@@ -1566,6 +1568,96 @@ pub async fn apply_service_interface_templates<
     apply_service_interface_templates_with_scope(repo, namespace, interfaces, "", None).await
 }
 
+async fn wait_for_service_interface_deletion<
+    R: crate::repository::DpuServiceInterfaceRepository,
+>(
+    repo: &R,
+    name: &str,
+    namespace: &str,
+) -> Result<(), DpfError> {
+    tokio::time::timeout(SERVICE_INTERFACE_DELETE_TIMEOUT, async {
+        loop {
+            if crate::repository::DpuServiceInterfaceRepository::get(repo, name, namespace)
+                .await?
+                .is_none()
+            {
+                return Ok(());
+            }
+            tokio::time::sleep(SERVICE_INTERFACE_DELETE_POLL_INTERVAL).await;
+        }
+    })
+    .await
+    .map_err(|_| {
+        DpfError::timeout(
+            "DPUServiceInterface deletion",
+            format!("{namespace}/{name} still exists after {SERVICE_INTERFACE_DELETE_TIMEOUT:?}"),
+        )
+    })?
+}
+
+async fn delete_stale_legacy_service_interfaces<
+    R: crate::repository::DpuServiceInterfaceRepository,
+>(
+    repo: &R,
+    namespace: &str,
+) -> Result<(), crate::error::DpfError> {
+    let mut live_interfaces =
+        crate::repository::DpuServiceInterfaceRepository::list(repo, namespace).await?;
+    live_interfaces.sort_by(|left, right| left.metadata.name.cmp(&right.metadata.name));
+    for live in live_interfaces {
+        let Some(name) = live.metadata.name.clone() else {
+            continue;
+        };
+        if live.spec.template.spec.node_selector.is_some() {
+            continue;
+        }
+
+        tracing::info!(
+            namespace,
+            service_interface = name,
+            "Deleting stale legacy unscoped DPUServiceInterface during scoped migration"
+        );
+        crate::repository::DpuServiceInterfaceRepository::delete(repo, &name, namespace).await?;
+        wait_for_service_interface_deletion(repo, &name, namespace).await?;
+    }
+
+    Ok(())
+}
+
+async fn delete_stale_scoped_service_interfaces<
+    R: crate::repository::DpuServiceInterfaceRepository,
+>(
+    repo: &R,
+    namespace: &str,
+) -> Result<(), crate::error::DpfError> {
+    let mut live_interfaces =
+        crate::repository::DpuServiceInterfaceRepository::list(repo, namespace).await?;
+    live_interfaces.sort_by(|left, right| left.metadata.name.cmp(&right.metadata.name));
+    for live in live_interfaces {
+        let Some(name) = live.metadata.name.clone() else {
+            continue;
+        };
+        if !["bf3", "bf4", "astra"]
+            .iter()
+            .any(|suffix| name.ends_with(&format!("-{suffix}")))
+        {
+            continue;
+        }
+        if live.spec.template.spec.node_selector.is_none() {
+            continue;
+        }
+        tracing::info!(
+            namespace,
+            service_interface = name,
+            "Deleting stale scoped DPUServiceInterface during legacy unscoped migration"
+        );
+        crate::repository::DpuServiceInterfaceRepository::delete(repo, &name, namespace).await?;
+        wait_for_service_interface_deletion(repo, &name, namespace).await?;
+    }
+
+    Ok(())
+}
+
 async fn create_flavor_services_and_deployment<
     R: DpuServiceTemplateRepository
         + DpuServiceConfigurationRepository
@@ -1616,9 +1708,24 @@ async fn create_flavor_services_and_deployment<
     // management-plane DPUNode and DPUDeployment selection. Reusing them here matches zero remote
     // Nodes and prevents DPF from instantiating concrete ServiceInterfaces.
     //
-    // Changing either names or selectors triggers DPF reconciliation and must remain an explicit
-    // operator-controlled migration.
+    // Changing either names or selectors triggers DPF reconciliation. Initialization deletes
+    // resources from the opposite mode so stale templates do not remain active beside the
+    // current generation.
     //
+    // A legacy interface matches every remote DPU-cluster Node. Wait for resources from the
+    // opposite mode to be deleted before creating their replacements, so generations do not
+    // overlap during a namespace-wide transition.
+    let cleanup_result = if !config.deployment_scoped_service_interfaces {
+        delete_stale_scoped_service_interfaces(repo, namespace).await
+    } else {
+        delete_stale_legacy_service_interfaces(repo, namespace).await
+    };
+    cleanup_result.map_err(|error| {
+        DpfError::InvalidState(format!(
+            "failed to remove the previous DPUServiceInterface generation ({error}); resolve the deletion failure, then retry initialization. The previous generation may be only partially removed"
+        ))
+    })?;
+
     // Patch CRs require their peer bridge, so preserve flavor creation before interface templates.
     apply_service_interface_templates_with_scope(
         repo,
@@ -1627,8 +1734,12 @@ async fn create_flavor_services_and_deployment<
         interface_suffix,
         dpu_cluster_node_labels.as_ref(),
     )
-    .await?;
-
+    .await
+    .map_err(|error| {
+        DpfError::InvalidState(format!(
+            "failed to create replacement DPUServiceInterfaces ({error}); resolve the apply failure, then retry initialization. The previous generation has been removed and replacements may be only partially created"
+        ))
+    })?;
     // Each deployment gets its own service/NAD CRs (suffixed by deployment type)
     // so BF3 and BF4 do not overwrite each other's Helm values/versions in the
     // shared namespace. `nad_rename` maps each deployment-local NAD name to its