Skip to content

usage.md

Latest commit

 

History

History
215 lines (145 loc) · 12 KB

usage.md

File metadata and controls

215 lines (145 loc) · 12 KB
Raw

Usage and Validation

Run a sync from the Forward Sync detail page. The plugin executes the enabled NetBox models through the configured NQE maps, records failures as Forward Ingestion Issues, and applies the selected execution backend. The default backend stages changes in native NetBox Branching shards for review and merge. The optional fast bootstrap backend writes directly after validation for trusted large baselines.

Version v0.9.4.1.1 is the point where the architecture is split into explicit execution-ledger, health, and support-reporting surfaces while keeping the shared 4.5/4.6 branch line. In practice, that means the UI still looks like a normal NetBox sync flow, but long runs now have clearer run/step state, better troubleshooting export, and less ambiguity around branch planning and merge progress.

Self-Test Workflow

Use this flow to validate a new installation from the UI.

1. Create A Source

Open Plugins > Forward > Sources > Add.

Fill in:

  • Type: Forward SaaS for https://fwd.app, or Custom Forward deployment for another URL
  • Username
  • Password
  • Network

Important:

  • Network will remain empty until the Forward account has both username and password configured.

Expected result:

  • The form loads without errors.
  • The Network field populates from the authenticated Forward tenant.
  • Saving the source returns you to the source detail page.

Forward Source

2. Review Built-In NQE Maps

Open Plugins > Forward > NQE Maps.

Expected result:

  • The seeded built-in maps are present.
  • Each built-in map shows a NetBox Model, execution mode, and enabled state.
  • Opening a built-in map displays either the shipped raw NQE text or the configured Query ID.

Forward NQE Maps

3. Create A Sync

Open Plugins > Forward > Syncs > Add.

Recommended first pass:

  • Select the source you just created.
  • Optionally select a Drift policy if you want validation to block unsafe changes before any branch is created.
  • Leave Snapshot at latestProcessed.
  • Leave Execution backend at Branching for reviewable syncs.
  • Leave Max changes per branch at 10000 unless local Branching guidance says otherwise.
  • Keep the default model selection enabled.
  • Leave Auto merge enabled to advance through all native Branching shards automatically.
  • Disable Auto merge when you want to review and manually merge each shard before clicking Continue Ingestion.
  • Select Fast bootstrap only for a trusted initial baseline that is too large to stage in Branching. After it completes, edit the sync back to Branching for steady-state diff runs.

See Initial Baseline Strategy for the decision table.

Expected result:

  • The sync saves cleanly.
  • The sync detail page shows the selected source, network, snapshot selection, drift policy, latest validation result, and enabled model list.

Forward Sync Detail

4. Validate The Sync

From the sync detail page, click Validate.

Expected result:

  • A Forward Validation Run is created without creating NetBox Branching branches.
  • The validation run records the resolved snapshot, optional baseline snapshot, per-model query results, drift summary, and blocking reasons.
  • If the selected drift policy blocks the run, the sync can be corrected before staging NetBox changes.
  • If you intentionally accept the blocked result, open the validation run and use Force allow to record the override reason and reviewer before rerunning the sync.

Forward Validation Detail

5. Run An Adhoc Ingestion

From the sync detail page, click Adhoc Ingestion.

Expected result:

  • A validation run is recorded before branch creation.
  • A new Forward Ingestion is created.
  • With the Branching backend, the sync status progresses into the branch-backed staging flow.
  • With the Branching backend, the sync creates one ingestion per shard, and each shard links to its native NetBox Branching branch.
  • With the Fast bootstrap backend, one branchless ingestion is created and rows are written directly after validation.
  • Fast bootstrap statistics and the ingestion Changes tab are backed by NetBox ObjectChange records for that direct-write request, not by Branching ChangeDiff rows.
  • The ingestion records both the selected snapshot mode and the resolved snapshot ID used for NQE execution.
  • The ingestion links to the validation run and persists per-model query execution results.
  • If Auto merge is disabled, the sync pauses after the current shard reaches Ready to merge.
  • After a successful fast-bootstrap baseline, change Execution backend back to Branching so later latestProcessed runs can use reviewable diffs.

6. Review The Ingestion

Open the ingestion detail page and inspect:

  • status and timestamps
  • snapshot selection and resolved snapshot ID
  • snapshot state and processed time
  • Forward snapshot metrics
  • model results and validation status
  • ingestion issues
  • change diff when the ingestion used Branching
  • branch linkage when the ingestion used Branching

Expected result:

  • The ingestion detail page loads successfully.
  • The ingestion shows the snapshot actually used for NQE execution.
  • The ingestion shows Forward snapshot metrics for the selected snapshot when Forward returns them.
  • The ingestion shows per-model execution mode, row count, delete count, runtime, and shard metadata when available.
  • Issues is empty or contains actionable query/persistence errors.
  • The change diff represents the staged NetBox changes for review.

Forward Ingestions

Forward Ingestion Detail

7. Confirm The Merged Branches

With Auto merge enabled, Branching syncs merge each native Branching shard before the next shard runs. With Auto merge disabled, review and merge the current shard, then click Continue Ingestion on the sync to stage the next shard. Fast bootstrap syncs do not create branches to merge.

Expected result:

  • Each Branching shard branch is marked merged.
  • The synced objects are visible in standard NetBox object views.

What To Check After A Successful Test

  • Sites, devices, interfaces, prefixes, and the other selected models exist in NetBox.
  • The latest ingestion has no unresolved issues.
  • The latest ingestion shows the expected snapshot selector, resolved snapshot ID, and Forward metrics.
  • The branch diff matches the expected object additions and updates.
  • The source and sync statuses are back in a healthy state.

CLI Smoke Validation

For a repeatable live smoke run outside GitHub Actions, use the bundled management command through the local invoke task.

Set the required environment variables:

export FORWARD_SMOKE_USERNAME='your-forward-username'
export FORWARD_SMOKE_PASSWORD='your-forward-password'
export FORWARD_SMOKE_NETWORK_ID='your-network-id'

Run the smoke sync:

invoke forward_netbox.smoke-sync

Optional knobs:

  • FORWARD_SMOKE_URL defaults to https://fwd.app
  • FORWARD_SMOKE_SNAPSHOT_ID defaults to latestProcessed
  • FORWARD_SMOKE_MODELS accepts a comma-separated list of enabled NetBox models
  • invoke forward_netbox.smoke-sync --validate-only resolves the source/network/snapshot and executes the selected queries without creating an ingestion
  • --query-limit limits rows fetched per query during --validate-only; normal syncs page through the full NQE result set
  • invoke forward_netbox.smoke-sync --plan-only --max-changes-per-branch 10000 prints the native NetBox Branching shard plan for large baselines
  • invoke forward_netbox.smoke-sync --max-changes-per-branch 10000 stages and merges large baselines in multiple native branches
  • max_changes_per_branch is treated as a guideline during execution: shards up to 5% over the configured value are accepted; larger overruns are auto-split and retried
  • invoke forward_netbox.smoke-sync --no-auto-merge --max-changes-per-branch 10000 stages one native Branching shard and pauses for review
  • invoke forward_netbox.smoke-sync --execution-backend fast_bootstrap runs the trusted direct-write baseline backend after validation
  • invoke forward_netbox.smoke-sync --enable-bulk-orm enables the opt-in bulk ORM apply engine for the parity-tested model set: dcim.site, dcim.manufacturer, dcim.devicerole, dcim.platform, dcim.devicetype, ipam.vrf, and ipam.vlan. The adapter path remains the default for all models.

The normal UI/API Run Sync path uses the same execution backends exposed on the sync form. Use the command-line smoke sync when you need explicit plan-only output, a targeted model subset, or a timed local baseline run.

Optional Module Import Readiness

The built-in Forward Modules map is enabled by default. The native dcim.module import path is beta in v0.6.x: use it when you want chassis modules, line cards, supervisors, fabric modules, or routing engines modeled as native NetBox dcim.module objects instead of generic inventory items, and review the staged branch carefully before merging.

NetBox modules require matching module bays on the device. When dcim.module is enabled, module rows whose bays do not already exist are skipped with a non-blocking warning. This keeps Branching merges deterministic and avoids creating module-bay side effects inside the module shard.

The simplest path is the UI: on the sync detail page click Module Readiness to see how many bays are missing, then Create missing module bays to create them directly in NetBox (no CSV, no shell). Re-run the sync afterward and the modules land in their bays. The CLI helper below does the same check and emits a CSV for operators who prefer to import it manually. Either way, do this before enabling the model:

!!! note "Why module bays must be imported out-of-band" dcim.modulebay is an MPTT (nested) model. NetBox Branching cannot create module bays during a merge — a module bay created inside a branch fails to apply to main with Save with update_fields did not affect any rows. This is a NetBox limitation, not a data error. So module bays are imported directly into NetBox (out of band) via the readiness CSV below, not synced through a branch. If a sync still creates new devices whose device type defines module-bay templates, NetBox auto-instantiates those bays inside the branch and they cannot merge; the plugin collapses every such failure for an ingestion into a single dcim.modulebay merge issue (ModuleBayMergeUnsupported) that points back to this readiness workflow, rather than one issue per bay. Device and interface sync are unaffected.

python manage.py forward_module_readiness --sync-name "Forward Sync"

Local Docker shortcut:

invoke forward_netbox.module-readiness --sync-name "Forward Sync"

Expected result:

  • the helper runs the module NQE map through the normal Forward API path
  • it compares every (device, module_bay) result to existing NetBox module bays
  • it writes summary.json and netbox-module-bays.csv
  • import netbox-module-bays.csv through the native NetBox Module Bays import UI, then rerun the helper
  • enable dcim.module only after the helper reports zero missing devices and zero missing module bays

After the readiness helper reports zero missing devices, enable the dcim.module model and the Forward Modules NQE map for the sync. SFPs and other optics remain in the inventory-item path by default. When module sync is enabled, matching generic inventory rows for module-native component classes are removed during inventory ingestion so the same hardware is not represented twice.

Expected result:

  • the command creates or updates a disposable smoke Source and Sync
  • the sync resolves the selected snapshot and runs a real ingestion
  • the sync records validation and per-model execution metadata before branch staging
  • the command exits non-zero if the sync fails or any ingestion issues are recorded
  • in --validate-only mode, the command prints per-model query execution mode, row count, and runtime and exits non-zero on any query failure