fix: endpoint import on tag-scoped syncs + tag merge name collision (2.4.4)

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 Generated from the README compatibility table by `scripts/gen_changelog.py`. Do not edit by hand.
 
+## v2.4.4
+
+Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set (validated live: 355 Avocent endpoints import under a tag-scoped sync). Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create.
+
 ## v2.4.3
 
 Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change.

README.md

@@ -66,7 +66,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f
 
 | Plugin Release | NetBox Version | Status |
 | --- | --- | --- |
-| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. |
+| `v2.4.4` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set (validated live: 355 Avocent endpoints import under a tag-scoped sync). Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. |
+| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. |
 | `v2.4.2` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing (validated live: 355 Avocent endpoints import; `Mgmt_*` tags sync). Adds a **Publish Bundled Queries** button on the sync Health page (beside Refresh Query IDs) and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (they double-apply rows for the same model and churn — enable one). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. |
 | `v2.4.1` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. |
 | `v2.4.0` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. |

docs/01_User_Guide/README.md

@@ -20,14 +20,14 @@ pip install forward-netbox
 Alternatively, install a specific wheel or source archive from GitHub Releases:
 
 ```bash
-pip install /path/to/forward_netbox-2.4.3-py3-none-any.whl
-pip install /path/to/forward_netbox-2.4.3.tar.gz
+pip install /path/to/forward_netbox-2.4.4-py3-none-any.whl
+pip install /path/to/forward_netbox-2.4.4.tar.gz
 ```
 
 If you mirror the package into a private Python index, pin the same release version:
 
 ```bash
-pip install forward-netbox==2.4.3
+pip install forward-netbox==2.4.4
 ```
 
 ## Release Compatibility
@@ -39,7 +39,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f
 
 | Plugin Release | NetBox Version | Status |
 | --- | --- | --- |
-| `v2.4.3` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. |
+| `v2.4.4` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set. Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. |
+| `v2.4.3` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. |
 | `v2.4.2` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing. Adds a **Publish Bundled Queries** button on the sync Health page and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (duplicate-emission churn). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. |
 | `v2.4.1` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. |
 | `v2.4.0` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. |

docs/03_Plans/active/2026-07-09-endpoint-scope-and-tag-merge.md

@@ -0,0 +1,86 @@
+# Fix: endpoint import on tag-scoped syncs + tag merge name collision (2.4.4)
+
+## Goal
+
+Make opt-in SNMP-endpoint import (`sync_endpoints`) actually work on syncs that
+use a device-tag include scope, and stop the merge-phase
+`Tag with this Name already exists` ingestion issues.
+
+## Constraints
+
+- No schema or migration changes; drop-in from 2.4.3.
+- Endpoint import stays additive: a failed endpoint probe must never fail a
+  scoped sync.
+- Local scope filtering and out-of-scope prune must not delete imported
+  endpoints on later scoped syncs.
+- Never commit the ADP network id / token; validate live via the local source.
+
+## Touched Surfaces
+
+- `forward_netbox/utilities/query_fetch_execution.py`
+  (`_resolve_scoped_tag_scope`, new `_resolve_scoped_endpoint_names`,
+  `resolve_context` cache keys, `_context_artifact_descriptor` v3)
+- `forward_netbox/queries/forward_devices.nqe` and
+  `forward_devices_with_netbox_aliases.nqe` (endpoint branch include-scope
+  removal; exclude tags kept)
+- `forward_netbox/utilities/bulk_merge.py` (`_flush` extras.tag name/slug
+  coalesce)
+- Tests: `test_endpoints_import.py`, `test_bulk_merge.py`
+
+## Approach
+
+Two independent defects made tag-scoped endpoint import impossible:
+
+1. Query-side: the endpoint union branch required endpoints to carry the
+   device include tags. Endpoints rarely share device scoping tags (they scope
+   the modeled-device universe), so every endpoint was filtered in Forward.
+   The endpoint branch now applies exclude tags only.
+2. Plugin-side: `scoped_device_names` was resolved from `network.devices`
+   only, so the local scope filter (`_apply_device_tag_scope`) dropped every
+   endpoint row unconditionally — and prune-out-of-scope would emit them as
+   deletes. With `sync_endpoints` on, a second probe over `network.endpoints`
+   (exclude tags honored, warn-and-skip on failure) unions endpoint names into
+   the scoped set, so the local filter and prune keep endpoint rows. The
+   context memo key and the shared context artifact (v3) include the toggle so
+   a cached scope without endpoint names is never reused.
+
+Merge fix: while a merge applies device UPDATEs (ordered before CREATEs),
+netbox_branching sets device tags by name, get_or_creating the tag on main with
+a new pk; the branch's tag CREATE then violated the unique name constraint and
+surfaced as a ValidationError issue. `_flush` now treats a same-named or
+same-slug main-side `extras.tag` as already merged (skip, counted as applied).
+
+## Validation
+
+Live against the ADP demo network, reproducing the design partner's scenario
+(include tags endpoints don't carry + `sync_endpoints` on): both device queries
+still emit 355 Avocent/console endpoints with include params unused; the scope
+union adds 688 endpoint names; `_apply_device_tag_scope` keeps an endpoint row
+while dropping an out-of-scope device. Unit tests cover the scope union, the
+exclude-tag probe, probe-failure softness, the NQE include-scope removal, and a
+real branch-merge tag name collision (skips, no fallback, no issue). Full suite
++ lint + harness green.
+
+## Rollback
+
+Revert the branch — pure Python + query text + tests; no data migration. The
+context artifact version bump only invalidates a transient cache.
+
+## Decision Log
+
+- Include scope intentionally does NOT gate endpoints: requiring shared tags
+  silently broke the feature for every scoped source; exclude tags remain as
+  the targeted safety valve. Documented in the query comments and README.
+- Endpoint names are unioned into `scoped_device_names` rather than exempting
+  endpoint rows in `_apply_device_tag_scope`, so prune and shard-key pushdown
+  stay consistent without changing row shapes.
+- The tag coalesce lives in the bulk `_flush` path (skip before build), not the
+  per-object fallback, so the batch never hits the IntegrityError rollback.
+- Endpoint-probe failure disables endpoint emission for the run (and skips the
+  context-artifact save) rather than proceeding with an endpoint-less scope:
+  otherwise the query would still emit endpoint rows, the local filter would
+  drop them, and prune-out-of-scope would delete previously imported endpoints
+  (adversarial-review finding).
+- A skipped same-named tag create may carry different non-unique attrs
+  (color/description); these converge on the next sync via the apply-time
+  coalesce-by-name UPDATE — accepted over adding a merge-time write path.

docs/README.md

@@ -13,7 +13,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f
 
 | Plugin Release | NetBox Version | Status |
 | --- | --- | --- |
-| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. |
+| `v2.4.4` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set. Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. |
+| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. |
 | `v2.4.2` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing. Adds a **Publish Bundled Queries** button on the sync Health page and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (duplicate-emission churn). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. |
 | `v2.4.1` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. |
 | `v2.4.0` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. |

forward_netbox/__init__.py

@@ -8,7 +8,7 @@ class NetboxForwardConfig(PluginConfig):
     name = "forward_netbox"
     verbose_name = "Forward"
     description = "Sync Forward data into NetBox using built-in NQE queries."
-    version = "2.4.3"
+    version = "2.4.4"
     base_url = "forward"
     min_version = "4.6.4"
 

forward_netbox/queries/forward_devices.nqe

@@ -44,15 +44,12 @@ foreach row in (
   (
     // SNMP endpoints (opt-in): generic SSH/SNMP devices Forward collects but
     // does not model. Generic identity from MIB-2; Avocent overlay by enterprise
-    // OID 10418. Scoped by the same device tags (endpoints carry tagNames).
+    // OID 10418. Device-tag INCLUDE scope narrows modeled devices only —
+    // endpoints rarely carry device scoping tags, so requiring them silently
+    // excluded every endpoint on scoped syncs. Exclude tags still apply.
     foreach endpoint in network.endpoints
     where sync_endpoints
     where !isEmpty(endpoint.snmpOutputs)
-    where isEmpty(device_tag_include_tags)
-      || (device_tag_include_match == "all"
-        && all(foreach tag in device_tag_include_tags select tag in endpoint.tagNames))
-      || (device_tag_include_match != "all"
-        && any(foreach tag in device_tag_include_tags select tag in endpoint.tagNames))
     where isEmpty(device_tag_exclude_tags)
       || !any(foreach tag in device_tag_exclude_tags select tag in endpoint.tagNames)
     let sysDescrOpt = max(foreach o in endpoint.snmpOutputs where o.requestedOid == "1.3.6.1.2.1.1.1" select max(foreach e in o.rawOidEntries select e.rawValue))

forward_netbox/queries/forward_devices_with_netbox_aliases.nqe

@@ -86,16 +86,13 @@ foreach row in (
   (
     // SNMP endpoints (opt-in): identical branch to forward_devices.nqe. Generic
     // SSH/SNMP devices Forward collects but does not model. Generic identity from
-    // MIB-2; Avocent overlay by enterprise OID 10418. Scoped by the same device
-    // tags (endpoints carry tagNames).
+    // MIB-2; Avocent overlay by enterprise OID 10418. Device-tag INCLUDE scope
+    // narrows modeled devices only — endpoints rarely carry device scoping tags,
+    // so requiring them silently excluded every endpoint on scoped syncs.
+    // Exclude tags still apply.
     foreach endpoint in network.endpoints
     where sync_endpoints
     where !isEmpty(endpoint.snmpOutputs)
-    where isEmpty(device_tag_include_tags)
-      || (device_tag_include_match == "all"
-        && all(foreach tag in device_tag_include_tags select tag in endpoint.tagNames))
-      || (device_tag_include_match != "all"
-        && any(foreach tag in device_tag_include_tags select tag in endpoint.tagNames))
     where isEmpty(device_tag_exclude_tags)
       || !any(foreach tag in device_tag_exclude_tags select tag in endpoint.tagNames)
     let sysDescrOpt = max(foreach o in endpoint.snmpOutputs where o.requestedOid == "1.3.6.1.2.1.1.1" select max(foreach e in o.rawOidEntries select e.rawValue))