Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

Generated from the README compatibility table by `scripts/gen_changelog.py`. Do not edit by hand.

## v2.5.0

Feature: optional **netbox-dlm** (Device Lifecycle Management) integration — three new opt-in NQE maps sync Forward's end-of-life analysis into the netbox-dlm plugin: OS software versions with vendor EOL dates per (platform, version), hardware end-of-life notices per device type (Cisco/Palo Alto/Fortinet part support), and each device's running software version. Disabled by default; requires the netbox-dlm plugin (run `makemigrations netbox_dlm && migrate` after installing it — it ships no migrations). Fix: syncs no longer crash mid-provision when an installed plugin's migrations were never applied (`relation ... does not exist`) — a preflight now fails in seconds with the app name and remedy, and a new **Database tables** Health check surfaces the gap before you sync.

## v2.4.5

Fix: sync no longer crashes on netbox-branching **1.1.1** (`SquashMergeStrategy has no attribute '_split_bidirectional_cycles'` — 1.1.1 removed that internal helper; the bidirectional-cycle split is now built into the plugin and the dependency is bounded to `<1.2`). Also fixes SNMP-endpoint rows failing validation: the bundled endpoint query branches now clamp sysDescr-derived `device_type` to NetBox's 100-char limit (`substring`) and guard empty slugs — the fix lives in the NQE queries (the source of truth), so **Publish Bundled Queries** again after upgrading (fixes the `Ensure this value has at most 100 characters` rejects and the `At least one coalesce lookup must be provided` error).
Expand Down
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ On top of the import it adds scope control (sync only devices carrying chosen Fo
- **It is not a source of truth for Forward configuration or intent** — it
populates NetBox from Forward's collected snapshot, nothing more.
- **It does not require** the optional `netbox-routing` / `netbox-peering-manager`
plugins unless you enable the beta BGP/OSPF maps.
/ `netbox-dlm` plugins unless you enable the beta BGP/OSPF or device-lifecycle
maps.

## Screenshots

Expand Down Expand Up @@ -66,7 +67,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f

| Plugin Release | NetBox Version | Status |
| --- | --- | --- |
| `v2.4.5` | `4.6.4` required; needs netbox-branching `1.1.0` – `1.1.x` | Current release; Fix: sync no longer crashes on netbox-branching **1.1.1** (`SquashMergeStrategy has no attribute '_split_bidirectional_cycles'` — 1.1.1 removed that internal helper; the bidirectional-cycle split is now built into the plugin and the dependency is bounded to `<1.2`). Also fixes SNMP-endpoint rows failing validation: the bundled endpoint query branches now clamp sysDescr-derived `device_type` to NetBox's 100-char limit (`substring`) and guard empty slugs — the fix lives in the NQE queries (the source of truth), so **Publish Bundled Queries** again after upgrading (fixes the `Ensure this value has at most 100 characters` rejects and the `At least one coalesce lookup must be provided` error). |
| `v2.5.0` | `4.6.4` required; needs netbox-branching `1.1.0` – `1.1.x` | Current release; Feature: optional **netbox-dlm** (Device Lifecycle Management) integration — three new opt-in NQE maps sync Forward's end-of-life analysis into the netbox-dlm plugin: OS software versions with vendor EOL dates per (platform, version), hardware end-of-life notices per device type (Cisco/Palo Alto/Fortinet part support), and each device's running software version. Disabled by default; requires the netbox-dlm plugin (run `makemigrations netbox_dlm && migrate` after installing it — it ships no migrations). Fix: syncs no longer crash mid-provision when an installed plugin's migrations were never applied (`relation ... does not exist`) — a preflight now fails in seconds with the app name and remedy, and a new **Database tables** Health check surfaces the gap before you sync. |
| `v2.4.5` | `4.6.4` required; needs netbox-branching `1.1.0` – `1.1.x` | Superseded by `v2.5.0`; Fix: sync no longer crashes on netbox-branching **1.1.1** (`SquashMergeStrategy has no attribute '_split_bidirectional_cycles'` — 1.1.1 removed that internal helper; the bidirectional-cycle split is now built into the plugin and the dependency is bounded to `<1.2`). Also fixes SNMP-endpoint rows failing validation: the bundled endpoint query branches now clamp sysDescr-derived `device_type` to NetBox's 100-char limit (`substring`) and guard empty slugs — the fix lives in the NQE queries (the source of truth), so **Publish Bundled Queries** again after upgrading (fixes the `Ensure this value has at most 100 characters` rejects and the `At least one coalesce lookup must be provided` error). |
| `v2.4.4` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.5`; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set (validated live: 355 Avocent endpoints import under a tag-scoped sync). Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. |
| `v2.4.3` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure — it reads "nothing new syncs" and fires on any pinned map with the feature on, even after the query is fixed, because the Health page can't read a pinned query's contents. Reworded to a "Pinned — can't verify locally" heads-up that points at **Export Live Query Drift** to confirm (`source_matches_bundled`), instead of asserting failure. No behavior change. |
| `v2.4.2` | `4.6.4` required; needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing (validated live: 355 Avocent endpoints import; `Mgmt_*` tags sync). Adds a **Publish Bundled Queries** button on the sync Health page (beside Refresh Query IDs) and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (they double-apply rows for the same model and churn — enable one). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. |
Expand Down Expand Up @@ -201,6 +203,7 @@ use.
- Optional disabled NQE maps for NetBox Device Type Library alias matching through a Forward JSON data file
- Optional disabled NQE map for data-file-driven device feature tag rules
- Feature-flagged beta BGP and OSPF maps for optional `netbox-routing` and `netbox-peering-manager` deployments
- Beta device-lifecycle maps for optional `netbox-dlm` deployments: OS/hardware end-of-life dates and per-device running software from Forward's support analysis
- Snapshot-aware execution with `latestProcessed` or an explicit Forward snapshot per sync
- Ingestion records that preserve the selected snapshot mode, resolved snapshot ID, and Forward snapshot metrics
- Built-in coverage for:
Expand Down
9 changes: 5 additions & 4 deletions docs/01_User_Guide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,14 @@ pip install forward-netbox
Alternatively, install a specific wheel or source archive from GitHub Releases:

```bash
pip install /path/to/forward_netbox-2.4.5-py3-none-any.whl
pip install /path/to/forward_netbox-2.4.5.tar.gz
pip install /path/to/forward_netbox-2.5.0-py3-none-any.whl
pip install /path/to/forward_netbox-2.5.0.tar.gz
```

If you mirror the package into a private Python index, pin the same release version:

```bash
pip install forward-netbox==2.4.5
pip install forward-netbox==2.5.0
```

## Release Compatibility
Expand All @@ -39,7 +39,8 @@ Latest release requires NetBox `4.6.4` and `netbox-branching` `1.1.0+`. Expand f

| Plugin Release | NetBox Version | Status |
| --- | --- | --- |
| `v2.4.5` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0` – `1.1.x` | Current release; Fix: sync no longer crashes on netbox-branching **1.1.1** (`SquashMergeStrategy has no attribute '_split_bidirectional_cycles'` — 1.1.1 removed that internal helper; the bidirectional-cycle split is now built into the plugin and the dependency is bounded to `<1.2`). Also fixes SNMP-endpoint rows failing validation: the bundled endpoint query branches now clamp sysDescr-derived `device_type` to NetBox's 100-char limit (`substring`) and guard empty slugs — the fix lives in the NQE queries (the source of truth), so **Publish Bundled Queries** again after upgrading (fixes the `Ensure this value has at most 100 characters` rejects and the `At least one coalesce lookup must be provided` error). |
| `v2.5.0` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0` – `1.1.x` | Current release; Feature: optional **netbox-dlm** (Device Lifecycle Management) integration — three new opt-in NQE maps sync Forward's end-of-life analysis into the netbox-dlm plugin: OS software versions with vendor EOL dates per (platform, version), hardware end-of-life notices per device type (Cisco/Palo Alto/Fortinet part support), and each device's running software version. Disabled by default; requires the netbox-dlm plugin (run `makemigrations netbox_dlm && migrate` after installing it — it ships no migrations). Fix: syncs no longer crash mid-provision when an installed plugin's migrations were never applied (`relation ... does not exist`) — a preflight now fails in seconds with the app name and remedy, and a new **Database tables** Health check surfaces the gap before you sync. |
| `v2.4.5` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0` – `1.1.x` | Superseded by `v2.5.0`; Fix: sync no longer crashes on netbox-branching **1.1.1** (`SquashMergeStrategy has no attribute '_split_bidirectional_cycles'` — 1.1.1 removed that internal helper; the bidirectional-cycle split is now built into the plugin and the dependency is bounded to `<1.2`). Also fixes SNMP-endpoint rows failing validation: the bundled endpoint query branches now clamp sysDescr-derived `device_type` to NetBox's 100-char limit (`substring`) and guard empty slugs — the fix lives in the NQE queries (the source of truth), so **Publish Bundled Queries** again after upgrading (fixes the `Ensure this value has at most 100 characters` rejects and the `At least one coalesce lookup must be provided` error). |
| `v2.4.4` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.5`; Fix: SNMP-endpoint import now works on **tag-scoped** syncs — the device-tag include scope silently excluded every endpoint, both query-side and in the plugin's local scope filter (whose scoped-device set was built from modeled devices only, so endpoint rows were always dropped; with prune enabled they would even be deleted). Endpoint import now ignores the include scope (exclude tags still apply) and endpoint names join the scoped set. Also fixes the merge-phase `Tag with this Name already exists` issues: a same-named/same-slug tag already on main is now treated as merged instead of failing the branch's tag create. |
| `v2.4.3` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.4`; Fix: the pinned-query opt-in Health warning (2.4.1) over-claimed failure and fired even after the query was fixed (the Health page can't read a pinned query's contents). Reworded to a "Pinned — can't verify locally" heads-up pointing at Export Live Query Drift to confirm. No behavior change. |
| `v2.4.2` | `4.6.4` required (4.5.x dropped); needs netbox-branching `1.1.0+` | Superseded by `v2.4.3`; Fix: endpoint import (`sync_endpoints`) and device-tag sync (`sync_device_tags`) now work with the alias-aware and rules-aware query variants (`forward_devices_with_netbox_aliases`, `forward_device_feature_tags_with_rules`), not just the base queries — operators running the variants saw the toggles silently do nothing. Adds a **Publish Bundled Queries** button on the sync Health page and two Health warnings: when an opt-in feature is enabled but no enabled map provides it, and when a base query and its opt-in variant are both enabled (duplicate-emission churn). The alias-aware device query now emits the clean role name (e.g. `ROUTER`) to match the base query — expect a one-time role update on alias-mapped devices. |
Expand Down
3 changes: 3 additions & 0 deletions docs/02_Reference/apply-engine-model-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ These models remain adapter-required with explicit blocker codes:
- `netbox_cisco_aci.acitenant` | `plugin_model_dependencies`
- `netbox_cisco_aci.acivrf` | `plugin_model_dependencies`
- `netbox_peering_manager.peeringsession` | `plugin_model_dependencies`
- `netbox_dlm.devicesoftware` | `plugin_model_dependencies`
- `netbox_dlm.hardwarenotice` | `plugin_model_dependencies`
- `netbox_dlm.softwareversion` | `plugin_model_dependencies`
- `netbox_routing.bgpaddressfamily` | `plugin_model_dependencies`
- `netbox_routing.bgppeer` | `plugin_model_dependencies`
- `netbox_routing.bgppeeraddressfamily` | `plugin_model_dependencies`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Feature: optional netbox-dlm integration + provision table preflight (2.5.0)

## Goal

Stop syncs from crashing when an installed plugin's migrations were never
applied (field report: `relation "public.netbox_dlm_contract" does not exist`
mid-provision), and add an optional integration for the netbox-dlm Device
Lifecycle Management plugin backed by Forward's end-of-life analysis.

## Constraints

- netbox-dlm is NOT installed in CI or the dev runtime; every surface must
degrade gracefully when the plugin is absent (existing optional-plugin
pattern) and the new maps ship disabled.
- New model strings require a ForwardNQEMap choices migration (0032) — the
only schema change; no data migrations.
- NQE stays the source of truth: field-length clamping and identity shaping
live in the bundled queries, not the plugin.
- Never commit the ADP network id / token.

## Touched Surfaces

- `forward_netbox/utilities/branching.py` (`missing_branch_table_report`),
`single_branch_executor.py` (preflight), `health.py` (Database tables check)
- `forward_netbox/utilities/sync_dlm.py` (new adapter),
`sync_runner_adapters.py` + `sync.py` (dispatch wiring)
- `forward_netbox/choices.py` (+ migration 0032), `query_registry.py`,
`plugin_integrations/registry.py`, `apply_engine_decision.py`,
`branch_budget.py`, `sync_contracts.py`, `sync_primitives.py`,
`query_fetch_execution.py`, `sync_reporting.py`
- New queries: `forward_dlm_software_versions.nqe`,
`forward_dlm_hardware_notices.nqe`, `forward_dlm_device_software.nqe`
- Tests: `test_provision_preflight.py`, `test_dlm_integration.py`

## Approach

1. Preflight: Django registers ContentTypes for every installed app even when
its migrations never ran, so netbox_branching's provision runs
`CREATE TABLE branch.T (LIKE public.T)` for a missing table and the sync
dies with an opaque ProgrammingError. `missing_branch_table_report()` diffs
`get_tables_to_replicate()` against `connection.introspection.table_names()`
(the exact list provision replicates), maps missing tables to app labels,
and the executor raises an actionable SyncError before the fetch. A
fail-severity **Database tables** Health check surfaces the same report.
2. netbox-dlm integration (three opt-in maps, following the
netbox_routing/ACI optional-plugin pattern end to end): SoftwareVersion
from `device.platform.osSupport` per (platform, osVersion); HardwareNotice
from chassis `device.platform.components[].support` per device type;
DeviceSoftware one row per device (the adapter ensures the referenced
SoftwareVersion exists without overwriting announced dates). Registry
integration (`DLM_INTEGRATION`, adapter contract on
`forward_netbox.utilities.sync_dlm`), decision/budget/contract/identity
tables, and runner dispatch are wired for all three model strings.
3. Fixed a latent import-order bug the new models exposed:
`_fallback_bucket_key_family` was defined after the module-level shard
contract build in `branch_budget.py` and NameError'd for any supported
model without a structured shard contract.

## Validation

Live against the ADP demo network: 35 software-version rows (ISO dates,
announcement URLs), 48 hardware-notice rows, 4960 device-software rows; all
three bundled queries lint clean and run via raw NQE. Unit tests cover the
preflight (report, executor SyncError, Health check pass/fail/soft-fail) and
the DLM wiring (choices/registry/maps disabled-by-default/adapter contract/
runner dispatch/query structure). Full suite + lint + harness green.

## Rollback

Revert the branch and roll back migration 0032 (choices-only alter). No data
migrations; the new maps are seeded disabled and can be deleted.

## Decision Log

- Ship the DLM maps disabled: the plugin is alpha (0.1.0), ships no
migrations, and the integration is a beta surface — operators opt in per map.
- DeviceSoftware ensures a bare SoftwareVersion when the vendor has no EOL
announcement (create-only, never overwriting dates from the versions map),
so every device links even when Forward lacks announcement data.
- Forward exposes no end-of-sale date, so HardwareNotice.end_of_sale stays
null; end_of_support/lastSupportDate, end_of_security_patches/
lastVulnerabilityDate, end_of_sw_releases/lastMaintenanceDate.
- The preflight lives in the executor (not provision monkeypatching) and the
Health check never raises — diagnostics failures return None rather than
breaking the page.
Loading
Loading