diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d3639ec..1c70624c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ Generated from the README compatibility table by `scripts/gen_changelog.py`. Do not edit by hand. +## v2.4.4 + +Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set (validated live: 355 Avocent endpoints import under a tag-scoped sync). Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. + ## v2.4.3 Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. diff --git a/README.md b/README.md index da925801..f2d525d1 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f | Plugin Release | NetBox Version | Status | | --- | --- | --- | -| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. | +| `v2.4.4` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set (validated live: 355 Avocent endpoints import under a tag-scoped sync). Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. | +| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. | | `v2.4.2` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing (validated live: 355 Avocent endpoints import; `Mgmt_*` tags sync). Adds a **Publish Bundled Queries** button on the sync Health page (beside Refresh Query IDs) and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (they double-apply rows for the same model and churn — enable one). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. | | `v2.4.1` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. | | `v2.4.0` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. | diff --git a/docs/01_User_Guide/README.md b/docs/01_User_Guide/README.md index cd315a19..c2756a60 100644 --- a/docs/01_User_Guide/README.md +++ b/docs/01_User_Guide/README.md @@ -20,14 +20,14 @@ pip install forward-netbox Alternatively, install a specific wheel or source archive from GitHub Releases: ```bash -pip install /path/to/forward_netbox-2.4.3-py3-none-any.whl -pip install /path/to/forward_netbox-2.4.3.tar.gz +pip install /path/to/forward_netbox-2.4.4-py3-none-any.whl +pip install /path/to/forward_netbox-2.4.4.tar.gz ``` If you mirror the package into a private Python index, pin the same release version: ```bash -pip install forward-netbox==2.4.3 +pip install forward-netbox==2.4.4 ``` ## Release Compatibility @@ -39,7 +39,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f | Plugin Release | NetBox Version | Status | | --- | --- | --- | -| `v2.4.3` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. | +| `v2.4.4` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set. Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. | +| `v2.4.3` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. | | `v2.4.2` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing. Adds a **Publish Bundled Queries** button on the sync Health page and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (duplicate-emission churn). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. | | `v2.4.1` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. | | `v2.4.0` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. | diff --git a/docs/03_Plans/active/2026-07-09-endpoint-scope-and-tag-merge.md b/docs/03_Plans/active/2026-07-09-endpoint-scope-and-tag-merge.md new file mode 100644 index 00000000..d032240d --- /dev/null +++ b/docs/03_Plans/active/2026-07-09-endpoint-scope-and-tag-merge.md @@ -0,0 +1,86 @@ +# Fix: endpoint import on tag-scoped syncs + tag merge name collision (2.4.4) + +## Goal + +Make opt-in SNMP-endpoint import (`sync_endpoints`) actually work on syncs that +use a device-tag include scope, and stop the merge-phase +`Tag with this Name already exists` ingestion issues. + +## Constraints + +- No schema or migration changes; drop-in from 2.4.3. +- Endpoint import stays additive: a failed endpoint probe must never fail a + scoped sync. +- Local scope filtering and out-of-scope prune must not delete imported + endpoints on later scoped syncs. +- Never commit the ADP network id / token; validate live via the local source. + +## Touched Surfaces + +- `forward_netbox/utilities/query_fetch_execution.py` + (`_resolve_scoped_tag_scope`, new `_resolve_scoped_endpoint_names`, + `resolve_context` cache keys, `_context_artifact_descriptor` v3) +- `forward_netbox/queries/forward_devices.nqe` and + `forward_devices_with_netbox_aliases.nqe` (endpoint branch include-scope + removal; exclude tags kept) +- `forward_netbox/utilities/bulk_merge.py` (`_flush` extras.tag name/slug + coalesce) +- Tests: `test_endpoints_import.py`, `test_bulk_merge.py` + +## Approach + +Two independent defects made tag-scoped endpoint import impossible: + +1. Query-side: the endpoint union branch required endpoints to carry the + device include tags. Endpoints rarely share device scoping tags (they scope + the modeled-device universe), so every endpoint was filtered in Forward. + The endpoint branch now applies exclude tags only. +2. Plugin-side: `scoped_device_names` was resolved from `network.devices` + only, so the local scope filter (`_apply_device_tag_scope`) dropped every + endpoint row unconditionally — and prune-out-of-scope would emit them as + deletes. With `sync_endpoints` on, a second probe over `network.endpoints` + (exclude tags honored, warn-and-skip on failure) unions endpoint names into + the scoped set, so the local filter and prune keep endpoint rows. The + context memo key and the shared context artifact (v3) include the toggle so + a cached scope without endpoint names is never reused. + +Merge fix: while a merge applies device UPDATEs (ordered before CREATEs), +netbox_branching sets device tags by name, get_or_creating the tag on main with +a new pk; the branch's tag CREATE then violated the unique name constraint and +surfaced as a ValidationError issue. `_flush` now treats a same-named or +same-slug main-side `extras.tag` as already merged (skip, counted as applied). + +## Validation + +Live against the ADP demo network, reproducing the design partner's scenario +(include tags endpoints don't carry + `sync_endpoints` on): both device queries +still emit 355 Avocent/console endpoints with include params unused; the scope +union adds 688 endpoint names; `_apply_device_tag_scope` keeps an endpoint row +while dropping an out-of-scope device. Unit tests cover the scope union, the +exclude-tag probe, probe-failure softness, the NQE include-scope removal, and a +real branch-merge tag name collision (skips, no fallback, no issue). Full suite ++ lint + harness green. + +## Rollback + +Revert the branch — pure Python + query text + tests; no data migration. The +context artifact version bump only invalidates a transient cache. + +## Decision Log + +- Include scope intentionally does NOT gate endpoints: requiring shared tags + silently broke the feature for every scoped source; exclude tags remain as + the targeted safety valve. Documented in the query comments and README. +- Endpoint names are unioned into `scoped_device_names` rather than exempting + endpoint rows in `_apply_device_tag_scope`, so prune and shard-key pushdown + stay consistent without changing row shapes. +- The tag coalesce lives in the bulk `_flush` path (skip before build), not the + per-object fallback, so the batch never hits the IntegrityError rollback. +- Endpoint-probe failure disables endpoint emission for the run (and skips the + context-artifact save) rather than proceeding with an endpoint-less scope: + otherwise the query would still emit endpoint rows, the local filter would + drop them, and prune-out-of-scope would delete previously imported endpoints + (adversarial-review finding). +- A skipped same-named tag create may carry different non-unique attrs + (color/description); these converge on the next sync via the apply-time + coalesce-by-name UPDATE — accepted over adding a merge-time write path. diff --git a/docs/README.md b/docs/README.md index 953f8aa3..0416d8af 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f | Plugin Release | NetBox Version | Status | | --- | --- | --- | -| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. | +| `v2.4.4` | `4.6.4` required; needs netbox-branching `1.1.0+` | Current release; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set. Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. | +| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. | | `v2.4.2` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing. Adds a **Publish Bundled Queries** button on the sync Health page and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (duplicate-emission churn). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. | | `v2.4.1` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.2`; Fix: opt-in features (SNMP endpoint import, device-tag sync) silently did nothing on sources that run org-managed **pinned** Forward query IDs predating the feature — the sync Health page now raises an actionable warning instead of a silent badge. Remediation: publish the bundled queries to your Forward org folder (Overwrite on), then use Refresh Query IDs, then re-sync. | | `v2.4.0` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.1`; Fix: the "Import SNMP Endpoints as Devices" toggle now renders on the source form (the field shipped in 2.3.2 but was not in any fieldset, so it never showed), letting operators enable endpoint import from the GUI. | diff --git a/forward_netbox/__init__.py b/forward_netbox/__init__.py index f09315e7..46e6a48e 100644 --- a/forward_netbox/__init__.py +++ b/forward_netbox/__init__.py @@ -8,7 +8,7 @@ class NetboxForwardConfig(PluginConfig): name = "forward_netbox" verbose_name = "Forward" description = "Sync Forward data into NetBox using built-in NQE queries." - version = "2.4.3" + version = "2.4.4" base_url = "forward" min_version = "4.6.4" diff --git a/forward_netbox/queries/forward_devices.nqe b/forward_netbox/queries/forward_devices.nqe index eca3076d..1ca46804 100644 --- a/forward_netbox/queries/forward_devices.nqe +++ b/forward_netbox/queries/forward_devices.nqe @@ -44,15 +44,12 @@ foreach row in ( ( // SNMP endpoints (opt-in): generic SSH/SNMP devices Forward collects but // does not model. Generic identity from MIB-2; Avocent overlay by enterprise - // OID 10418. Scoped by the same device tags (endpoints carry tagNames). + // OID 10418. Device-tag INCLUDE scope narrows modeled devices only — + // endpoints rarely carry device scoping tags, so requiring them silently + // excluded every endpoint on scoped syncs. Exclude tags still apply. foreach endpoint in network.endpoints where sync_endpoints where !isEmpty(endpoint.snmpOutputs) - where isEmpty(device_tag_include_tags) - || (device_tag_include_match == "all" - && all(foreach tag in device_tag_include_tags select tag in endpoint.tagNames)) - || (device_tag_include_match != "all" - && any(foreach tag in device_tag_include_tags select tag in endpoint.tagNames)) where isEmpty(device_tag_exclude_tags) || !any(foreach tag in device_tag_exclude_tags select tag in endpoint.tagNames) let sysDescrOpt = max(foreach o in endpoint.snmpOutputs where o.requestedOid == "1.3.6.1.2.1.1.1" select max(foreach e in o.rawOidEntries select e.rawValue)) diff --git a/forward_netbox/queries/forward_devices_with_netbox_aliases.nqe b/forward_netbox/queries/forward_devices_with_netbox_aliases.nqe index 04fdb278..d9c441fd 100644 --- a/forward_netbox/queries/forward_devices_with_netbox_aliases.nqe +++ b/forward_netbox/queries/forward_devices_with_netbox_aliases.nqe @@ -86,16 +86,13 @@ foreach row in ( ( // SNMP endpoints (opt-in): identical branch to forward_devices.nqe. Generic // SSH/SNMP devices Forward collects but does not model. Generic identity from - // MIB-2; Avocent overlay by enterprise OID 10418. Scoped by the same device - // tags (endpoints carry tagNames). + // MIB-2; Avocent overlay by enterprise OID 10418. Device-tag INCLUDE scope + // narrows modeled devices only — endpoints rarely carry device scoping tags, + // so requiring them silently excluded every endpoint on scoped syncs. + // Exclude tags still apply. foreach endpoint in network.endpoints where sync_endpoints where !isEmpty(endpoint.snmpOutputs) - where isEmpty(device_tag_include_tags) - || (device_tag_include_match == "all" - && all(foreach tag in device_tag_include_tags select tag in endpoint.tagNames)) - || (device_tag_include_match != "all" - && any(foreach tag in device_tag_include_tags select tag in endpoint.tagNames)) where isEmpty(device_tag_exclude_tags) || !any(foreach tag in device_tag_exclude_tags select tag in endpoint.tagNames) let sysDescrOpt = max(foreach o in endpoint.snmpOutputs where o.requestedOid == "1.3.6.1.2.1.1.1" select max(foreach e in o.rawOidEntries select e.rawValue)) diff --git a/forward_netbox/tests/test_bulk_merge.py b/forward_netbox/tests/test_bulk_merge.py index dd27ac98..e8b56a8f 100644 --- a/forward_netbox/tests/test_bulk_merge.py +++ b/forward_netbox/tests/test_bulk_merge.py @@ -569,3 +569,54 @@ def test_splits_interface_lag_into_trailing_update(self): self.assertIn(upd_key, to_process) self.assertEqual(to_process[upd_key].postchange_data["lag"], 99) self.assertIn(cc.key, to_process[upd_key].depends_on) + + +class BulkMergeTagNameCollisionTest(TransactionTestCase): + """A branch tag CREATE that collides by NAME with a main-side tag skips. + + While a merge applies device UPDATEs (ordered before CREATEs), + netbox_branching sets device tags by name, get_or_creating the tag on main + with a new pk. The branch's tag CREATE then violated the unique name + constraint and surfaced as a ValidationError ingestion issue even though the + desired end state (tag exists) was already reached. + """ + + def setUp(self): + from django.contrib.auth import get_user_model + + self.user = get_user_model().objects.create_user(username="tag-merge-user") + self.request = RequestFactory().get(reverse("home")) + self.request.user = self.user + self.logger = logging.getLogger("forward_netbox.tests.bulk_merge") + + def test_tag_create_name_collision_skips_instead_of_failing(self): + from extras.models import Tag + + branch = provision_branch(user=self.user, name="Tag Collision") + with activate_branch(branch), event_tracking(self.request): + self.request.id = uuid.uuid4() + self.request.user = self.user + Tag.objects.create(name="Mgmt_Coll", slug="mgmt-coll") + + # Simulate the mid-merge name-based get_or_create on main: same name, + # different pk than the branch row. + Tag.objects.create(name="Mgmt_Coll", slug="mgmt-coll") + + changes = branch.get_unmerged_changes().order_by("time") + apply_one = Mock(return_value=False) # fallback would record an issue + applied, failed, _models = bulk_merge_changes( + branch, + changes, + self.request, + self.user, + self.logger, + apply_one=apply_one, + ) + + self.assertEqual(failed, 0) + # The colliding create must be skipped in the batch path, never routed + # to the per-object fallback (which would raise the ValidationError). + for call in apply_one.call_args_list: + collapsed = call.args[0] + self.assertNotEqual(collapsed.key[0], "extras.tag") + self.assertEqual(Tag.objects.filter(name="Mgmt_Coll").count(), 1) diff --git a/forward_netbox/tests/test_endpoints_import.py b/forward_netbox/tests/test_endpoints_import.py index e207735b..5e2b1137 100644 --- a/forward_netbox/tests/test_endpoints_import.py +++ b/forward_netbox/tests/test_endpoints_import.py @@ -135,3 +135,129 @@ def test_sync_endpoints_toggle_is_in_a_fieldset(self): "sync_endpoints field exists but is not in any FieldSet, so the " "form never renders the toggle.", ) + + +from forward_netbox.utilities.query_fetch_execution import ( # noqa: E402 + ForwardQueryFetcher, +) + + +class EndpointScopeUnionTest(TestCase): + """Tag-scoped syncs must keep opt-in endpoint rows, not silently drop them. + + scoped_device_names was built from network.devices only, so the local scope + filter removed every endpoint row (and prune would delete them). With + sync_endpoints on, endpoint names join the scoped set; exclude tags still + apply to the endpoint probe. + """ + + def _fetcher(self, client): + from unittest.mock import Mock + + return ForwardQueryFetcher(Mock(), client, Mock()) + + def test_scope_union_includes_endpoint_names(self): + from unittest.mock import Mock + + client = Mock() + client.run_nqe_query.side_effect = [ + [{"name": "dev-1", "site": "dc1", "tagNames": ["ACI"]}], # device scope + [{"name": "avocent-1"}, {"name": "avocent-2"}], # endpoint probe + ] + fetcher = self._fetcher(client) + names, _sites, _matched, failed = fetcher._resolve_scoped_tag_scope( + network_id="n", + snapshot_id="s", + include_tags=["ACI"], + exclude_tags=[], + include_match="any", + sync_endpoints=True, + ) + self.assertEqual(names, {"dev-1", "avocent-1", "avocent-2"}) + self.assertFalse(failed) + + def test_scope_without_endpoints_flag_is_unchanged(self): + from unittest.mock import Mock + + client = Mock() + client.run_nqe_query.return_value = [ + {"name": "dev-1", "site": "dc1", "tagNames": ["ACI"]} + ] + fetcher = self._fetcher(client) + names, _sites, _matched, failed = fetcher._resolve_scoped_tag_scope( + network_id="n", + snapshot_id="s", + include_tags=["ACI"], + exclude_tags=[], + include_match="any", + ) + self.assertEqual(names, {"dev-1"}) + self.assertFalse(failed) + client.run_nqe_query.assert_called_once() + + def test_endpoint_probe_honors_exclude_tags(self): + from unittest.mock import Mock + + client = Mock() + client.run_nqe_query.return_value = [{"name": "avocent-1"}] + fetcher = self._fetcher(client) + fetcher._resolve_scoped_endpoint_names( + network_id="n", snapshot_id="s", exclude_tags=["Decom"] + ) + query = client.run_nqe_query.call_args.kwargs["query"] + self.assertIn("network.endpoints", query) + self.assertIn('"Decom" in endpoint.tagNames', query) + + def test_endpoint_probe_failure_warns_and_returns_none(self): + from unittest.mock import Mock + + from forward_netbox.exceptions import ForwardClientError + + client = Mock() + client.run_nqe_query.side_effect = ForwardClientError("boom") + fetcher = self._fetcher(client) + names = fetcher._resolve_scoped_endpoint_names( + network_id="n", snapshot_id="s", exclude_tags=[] + ) + self.assertIsNone(names) + fetcher.logger.log_warning.assert_called_once() + + def test_endpoint_probe_failure_flags_scope_as_failed(self): + # If the probe fails but the query still emitted endpoint rows, the + # local filter would drop them — and prune would DELETE previously + # imported endpoints. The failure flag lets resolve_context disable + # endpoint emission for the run instead. + from unittest.mock import Mock + + from forward_netbox.exceptions import ForwardClientError + + client = Mock() + client.run_nqe_query.side_effect = [ + [{"name": "dev-1", "site": "dc1", "tagNames": ["ACI"]}], + ForwardClientError("boom"), + ] + fetcher = self._fetcher(client) + names, _sites, _matched, failed = fetcher._resolve_scoped_tag_scope( + network_id="n", + snapshot_id="s", + include_tags=["ACI"], + exclude_tags=[], + include_match="any", + sync_endpoints=True, + ) + self.assertEqual(names, {"dev-1"}) + self.assertTrue(failed) + + +class EndpointBranchIncludeScopeRemovalTest(SimpleTestCase): + """The endpoint branch must not gate on device INCLUDE tags (exclude only).""" + + def test_endpoint_branches_do_not_filter_include_tags(self): + for filename in ( + "forward_devices.nqe", + "forward_devices_with_netbox_aliases.nqe", + ): + src = _read_query(filename) + endpoint_branch = src.split("network.endpoints", 1)[1] + self.assertIn("device_tag_exclude_tags", endpoint_branch, filename) + self.assertNotIn("device_tag_include_tags", endpoint_branch, filename) diff --git a/forward_netbox/tests/test_scope_matched_tags.py b/forward_netbox/tests/test_scope_matched_tags.py index 509c6b6f..1325f0dd 100644 --- a/forward_netbox/tests/test_scope_matched_tags.py +++ b/forward_netbox/tests/test_scope_matched_tags.py @@ -41,7 +41,9 @@ def test_resolve_returns_per_device_intersection(self): {"name": "d3", "site": "s", "tagNames": ["TagA", "TagB"]}, {"name": "d4", "site": "s", "tagNames": ["TagX"]}, ] - names, sites, matched = self._fetcher(client)._resolve_scoped_tag_scope( + names, sites, matched, _failed = self._fetcher( + client + )._resolve_scoped_tag_scope( network_id="net-1", snapshot_id="snap", include_tags=["TagA", "TagB"], @@ -65,7 +67,7 @@ def test_resolve_no_tags_returns_empty_map(self): exclude_tags=[], include_match="any", ) - self.assertEqual(result, (set(), set(), {})) + self.assertEqual(result, (set(), set(), {}, False)) def test_context_as_dict_carries_full_matched_map(self): ctx = ForwardQueryContext( diff --git a/forward_netbox/utilities/bulk_merge.py b/forward_netbox/utilities/bulk_merge.py index 3bf50db2..8ab4c673 100644 --- a/forward_netbox/utilities/bulk_merge.py +++ b/forward_netbox/utilities/bulk_merge.py @@ -28,6 +28,7 @@ from django.db import DEFAULT_DB_ALIAS from django.db import IntegrityError from django.db import transaction +from django.db.models import Q from mptt.models import MPTTModel from netbox_branching.merge_strategies.squash import ActionType from netbox_branching.merge_strategies.squash import CollapsedChange @@ -398,6 +399,31 @@ def _flush(): existing = set( model_class.objects.filter(pk__in=all_pks).values_list("pk", flat=True) ) + # Tags collide by NAME, not pk: while the merge applies device UPDATEs + # (ordered before CREATEs), netbox_branching sets device tags by name, + # which get_or_creates the tag on main with a NEW pk. The branch's tag + # CREATE then violates the unique name/slug constraint. Treat a + # same-named (or same-slug) main-side tag as already merged. If the + # branch create carried different non-unique attrs (color/description), + # they converge on the next sync: its branch is provisioned from a main + # that has the tag, so the apply-time coalesce records an UPDATE. + existing_names: set[str] = set() + existing_slugs: set[str] = set() + if getattr(model_class._meta, "label_lower", "") == "extras.tag": + pending_names = [ + str((c.postchange_data or {}).get("name") or "") for c in pending + ] + pending_slugs = [ + str((c.postchange_data or {}).get("slug") or "") for c in pending + ] + name_filter = Q(name__in=[n for n in pending_names if n]) | Q( + slug__in=[s for s in pending_slugs if s] + ) + for name, slug in model_class.objects.filter(name_filter).values_list( + "name", "slug" + ): + existing_names.add(str(name)) + existing_slugs.add(str(slug)) skipped = 0 objects = [] built = [] # (collapsed, deserialized) for rows we actually create @@ -405,6 +431,14 @@ def _flush(): if collapsed_change.key[1] in existing: skipped += 1 continue + if existing_names or existing_slugs: + data = collapsed_change.postchange_data or {} + if ( + str(data.get("name") or "") in existing_names + or str(data.get("slug") or "") in existing_slugs + ): + skipped += 1 + continue try: deserialized = _deserialize( model_class, collapsed_change, change_logger diff --git a/forward_netbox/utilities/query_fetch_execution.py b/forward_netbox/utilities/query_fetch_execution.py index 69c20706..3272b297 100644 --- a/forward_netbox/utilities/query_fetch_execution.py +++ b/forward_netbox/utilities/query_fetch_execution.py @@ -392,6 +392,7 @@ def resolve_context(self, *, branch_run_state=None) -> ForwardQueryContext: tuple(exclude_tags), include_match, prune_out_of_scope, + sync_endpoints, branch_run_state.get("ingestion_id"), branch_run_state.get("pending_ingestion_id"), branch_run_state.get("current_ingestion_id"), @@ -406,6 +407,7 @@ def resolve_context(self, *, branch_run_state=None) -> ForwardQueryContext: include_tags=include_tags, exclude_tags=exclude_tags, include_match=include_match, + sync_endpoints=sync_endpoints, ) cached_context = self._load_context_artifact(context_artifact) if cached_context is None: @@ -428,21 +430,30 @@ def resolve_context(self, *, branch_run_state=None) -> ForwardQueryContext: scoped_device_names, scoped_site_names, scoped_matched_tags, + endpoint_scope_failed, ) = self._resolve_scoped_tag_scope( network_id=network_id, snapshot_id=snapshot_id, include_tags=include_tags, exclude_tags=exclude_tags, include_match=include_match, + sync_endpoints=sync_endpoints, ) - self._save_context_artifact( - context_artifact, - snapshot_info=snapshot_info, - snapshot_metrics=snapshot_metrics, - scoped_device_names=scoped_device_names, - scoped_site_names=scoped_site_names, - scoped_matched_tags=scoped_matched_tags, - ) + if endpoint_scope_failed: + # The scoped set carries no endpoint names, so emitting endpoint + # rows would get them dropped locally — and pruned as deletes. + # Disable endpoint emission for this run and do NOT persist the + # endpoint-less scope under the endpoints-on artifact key. + sync_endpoints = False + else: + self._save_context_artifact( + context_artifact, + snapshot_info=snapshot_info, + snapshot_metrics=snapshot_metrics, + scoped_device_names=scoped_device_names, + scoped_site_names=scoped_site_names, + scoped_matched_tags=scoped_matched_tags, + ) else: snapshot_info = dict(cached_context.get("snapshot_info") or {}) snapshot_metrics = dict(cached_context.get("snapshot_metrics") or {}) @@ -483,13 +494,16 @@ def _context_artifact_descriptor( include_tags: list[str], exclude_tags: list[str], include_match: str, + sync_endpoints: bool = False, ) -> dict[str, Any] | None: run = active_execution_run(self.sync) if run is None: return None artifact_run_id = f"shared-sync-{getattr(self.sync, 'pk', 'unknown')}" payload = { - "version": 2, + # v3: scoped_device_names now varies with the endpoint-import + # toggle, so the cache key must too. + "version": 3, "artifact_scope": "query_context", "cache_scope": "shared_sync", "sync_id": getattr(self.sync, "pk", None), @@ -499,6 +513,7 @@ def _context_artifact_descriptor( "device_tag_include": sorted(include_tags or []), "device_tag_exclude": sorted(exclude_tags or []), "device_tag_include_match": include_match or "any", + "sync_endpoints": bool(sync_endpoints), } return { "key": fetch_artifact_key(payload), @@ -575,9 +590,10 @@ def _resolve_scoped_tag_scope( include_tags: list[str], exclude_tags: list[str], include_match: str, - ) -> tuple[set[str], set[str], dict[str, list[str]]]: + sync_endpoints: bool = False, + ) -> tuple[set[str], set[str], dict[str, list[str]], bool]: if not include_tags and not exclude_tags: - return set(), set(), {} + return set(), set(), {}, False scope_where = build_device_tag_scope_where( include_tags, exclude_tags, include_match ) @@ -656,7 +672,77 @@ def _resolve_scoped_tag_scope( exclude_tags=exclude_tags, include_match=include_match, ) - return names, sites, matched_tags_by_device + endpoint_scope_failed = False + if sync_endpoints: + endpoint_names = self._resolve_scoped_endpoint_names( + network_id=network_id, + snapshot_id=snapshot_id, + exclude_tags=exclude_tags, + ) + if endpoint_names is None: + endpoint_scope_failed = True + else: + names |= endpoint_names + return names, sites, matched_tags_by_device, endpoint_scope_failed + + def _resolve_scoped_endpoint_names( + self, + *, + network_id: str, + snapshot_id: str, + exclude_tags: list[str], + ) -> set[str] | None: + """Names of SNMP endpoints that opt-in endpoint import will emit. + + Device-tag include scope narrows the modeled-device universe; SNMP + endpoints (e.g. Avocent console servers) rarely carry those device + scoping tags, so requiring them silently excludes every endpoint. When + endpoint import is enabled the endpoint names are unioned into the + scoped-device set so the local scope filter (and out-of-scope prune) + keeps their rows. Exclude tags still apply as the safety valve. + + Returns ``None`` when the probe fails. The caller must then disable + endpoint emission for the run entirely (not just proceed with an empty + set): the device query would still emit endpoint rows, the local scope + filter would drop them, and with prune-out-of-scope enabled the dropped + rows would be emitted as DELETES of previously imported endpoints. + """ + where = ["where !isEmpty(endpoint.snmpOutputs)"] + for tag in exclude_tags: + where.append(f"where !({_nqe_string_literal(tag)} in endpoint.tagNames)") + query = "\n".join( + [ + "foreach endpoint in network.endpoints", + *where, + "select { name: endpoint.name }", + ] + ) + try: + rows = self.client.run_nqe_query( + query=query, + network_id=network_id, + snapshot_id=snapshot_id, + fetch_all=True, + ) + except (ForwardClientError, ForwardConnectivityError, ForwardQueryError) as exc: + self.logger.log_warning( + "Forward endpoint scope probe failed; SNMP endpoint import is " + f"disabled for this run: {_safe_exception_summary(exc)}", + obj=self.sync, + ) + return None + endpoint_names = { + str(row.get("name") or "").strip() + for row in rows + if str(row.get("name") or "").strip() + } + if endpoint_names: + self.logger.log_info( + f"Added {len(endpoint_names)} SNMP endpoint(s) to the device tag " + "scope for opt-in endpoint import.", + obj=self.sync, + ) + return endpoint_names def _warn_if_scope_all_backfilled( self, diff --git a/pyproject.toml b/pyproject.toml index 266387a6..d0143a90 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "forward-netbox" -version = "2.4.3" +version = "2.4.4" description = "NetBox plugin to sync Forward data into NetBox via built-in NQE queries" authors = ["Craig Johnson "] license = "MIT"