Skip to content

test(fabric-api): Qualify the Looking Glass service on the lab - #1142

Merged
privateip merged 3 commits into
fix/lg-qualify-findingsfrom
fix/issue-1048
Oct 27, 2026
Merged

privateip merged 3 commits into
fix/lg-qualify-findingsfrom
fix/issue-1048

Conversation

@privateip

@privateip privateip commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The Looking Glass backend needs executed evidence on real fabric routers, against thresholds fixed in advance, before any cell turns the service on, and unit tests against fake nodes cannot supply it.

This adds an independent client, built only from the published service API, and a qualification suite that drives the service through it on the containerlab lab with faults, covering every row of the contract's acceptance matrix, including routing impact and cleanup.

A full run on the lab passed all 21 scenarios, with loaded BGP convergence at 1.00 times the control, and the qualification guide and an archived record describe how to repeat it and what it measured.

A lab pass does not qualify a staging or production cell, so the issue stays open until the suite runs there.

Stack

Second of two stacked PRs, on top of #1141, which fixes the four defects earlier runs of this suite found. Nothing here changes a shipped binary or manifest. The lab image build now records its commit, and rolling back means reverting this.

Test plan

  • The client builds and tests in its own module and imports nothing beyond the published service API
  • A full lab run passes every scenario with none skipped, and the lab's existing fabric checks pass after the rollback

Related to #1048

The Looking Glass service is consumed by a customer-facing service outside this repository, so its qualification needs a consumer that uses the published API the way that service will, not the gateway's own packages. examples/looking-glass-client is that consumer: a separate module whose path lies outside go.datum.net/galactic, so the compiler refuses any import of Galactic's internal packages, and which imports only api/fabric/service/v1 and api/fabric/v1.

It authenticates with a service certificate, pins the cell gateway's identity, and runs single diagnostics, workload specs (concurrency, bursts, per-project rates, slow or cancelling consumers) and arbitrary unary calls, printing one JSON record per attempt with its timing, terminal event and refusal details. It checks every stream it reads against the contract's event grammar and coverage rules.

task test:unit now also builds, vets and tests the module, and fails if it imports any Galactic package beyond the published service API. task lint lints it.

Related to #1048
The Looking Glass backend's acceptance requires executed evidence that its authorization, coverage, limits, cancellation and failover hold on real fabric routers, measured against thresholds fixed in advance, and that diagnostic load does not affect routing. Unit tests run the gateway against fake nodes and cannot show that.

tests/lgqualify drives the service on dfw's fabric routers through the independent client in examples/looking-glass-client, with fault injection. Its scenarios cover service authorization and capability compatibility; router coverage when routers are selected, omitted or failing; the public destination policy; the operator and FabricQuery interfaces with the service on; cell, project and destination bursts and sustained fairness; per-destination packet emission across every caller; cancellation, deadlines, the service switch and a blackholed node; leader crash, pause and API-server isolation; BGP convergence with and without diagnostic load; gateway and sidecar resources; the absence of targets and projects from routine logs and metric labels; and, last, that no injected fault outlives the run. Each threshold comes from the service contract and is fixed in thresholds_test.go. Each scenario writes its observations and verdict as JSON, and a skipped check fails rather than passes.

The lab gains what the run needs:
- deploy:looking-glass enables the service on dfw in the documented order, through overlays in resources/looking-glass, and issues the suite's service certificates; undeploy:looking-glass reverses it.
- tr4 announces a global IPv6 prefix for the off-fabric host, since the public probe policy refuses documentation space, and a sink prefix it drops silently.
- build:fabric-api stamps the commit into the image, which the gateway reports as its contract revision.

verify:looking-glass runs the suite. It restarts and pauses the gateway leader and resets one BGP session 80 times, so it is not part of verify.

Related to #1048
@privateip privateip changed the title test(fabric-api): Qualify the Looking Glass service on real fabric routers test(fabric-api): Qualify the Looking Glass service on the containerlab lab Oct 27, 2026
@privateip privateip changed the title test(fabric-api): Qualify the Looking Glass service on the containerlab lab test(fabric-api): Qualify the Looking Glass service on the lab Oct 27, 2026
Operators need to know how to qualify a Looking Glass build before switching the service on in a cell, what each scenario proves, and what a lab pass does not establish. docs/fabric-api/qualification.md describes the procedure: what the suite exercises, mapped to the contract's acceptance rows, how its thresholds are fixed, how to run and roll back on the lab, where the evidence goes, and the lab's limits. The fabric-api index, the architecture guide's test section and the operations guide's verification section link it.

The archive records the first passing run on the lab, with its measurements by acceptance row, the defects earlier runs found, and what the run does not establish, including the target platform.

Related to #1048
@privateip
privateip added this pull request to stack #1143 October 27, 2026 18:49
@privateip
privateip marked this pull request as ready for review October 27, 2026 19:01
@privateip
privateip requested a review from a team as a code owner October 27, 2026 19:01
@privateip

Copy link
Copy Markdown
Collaborator Author

Two independent reviews ran on this PR, then a second pass on the fixed head. The first pass held on an overstated title, a commit subject over 72 characters, a stale script comment and an output path the lab run never used. All four are fixed, and a dry run resolves the path to the repository's ignored bin directory. The second pass split, holding on the archive record's list of differences from the tested commit, which now names the script comment too.

The lab run recorded in the archive used commit 980cc91c. The published commits differ from it only in documentation, one gateway unit test, the lab task's output directory and a script comment, so the suite was not rerun on this head. A trial merge with current main, after #1140, has no conflicts.

CI is green on the current head. Merge the stack from this PR with the merge-async endpoint once both layers are approved, since GitHub does not allow auto-merge on stacked pull requests. This is related to #1048, which stays open until the suite runs on a staging cell.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Lab-only change: the client, suite, containerlab config and docs ship nothing to production, and the pass criteria are fixed thresholds that fail the run rather than pass vacuously. It is stacked on #1141 and must merge after it.

  • Stacked: the base is #1141's head (fix/lg-qualify-findings), so this diff is only the qualification work. It needs #1141's refusal policy, since the lab probe sink moved to global 2c0f:40::/64 and 11.1.42.0/24 because documentation space is now refused, and its client-auth certs for direct operator probes. Only docs/fabric-api/operations.md overlaps (one added paragraph, no conflict). Merge #1141 first and let GitHub retarget this PR.
  • Nothing ships: no .github/, config/ or containers/ file changes. Lab overlays live under deploy/containerlab/resources/looking-glass/ and only reference existing config/fabric-api components. The client is its own module under examples/, and test:looking-glass-client fails if it imports Galactic packages beyond the published service API.
  • Thresholds are consts in thresholds_test.go, fixed from the design doc. The scenarios fail on a miss, and the routing-impact scenario treats a control p95 whose bootstrap half-width is over 10% as a failure, not a pass. destinations asserts refusal with the exact reason for private, documentation, operator-allowed loopbacks, multicast and hostnames, and TestQualify calls t.Fatal if the client or binaries are missing.
  • Without LG_QUALIFY_KUBECONFIG the suite skips, so a bare go test ./... in CI passes without running it. That fits a 45-minute lab run, but a green check proves nothing about the Looking Glass. The archive benchmark file is the evidence of an actual run.
  • destinations checks the refusals but not that they cost no tokens, slots or packets, which is the pre-charge property #1141 adds.

Follow-ups:

  • Assert in destinations that the project and cell budgets are unchanged after a refused probe.
  • Add --deny-prefixes to the dfw gateway overlay so the lab exercises the gateway-side platform refusal and not only the special-use ranges.

@privateip
privateip merged commit 0e028cf into main Oct 27, 2026
23 checks passed
@privateip
privateip deleted the fix/issue-1048 branch October 27, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants