Repository navigation
test(fabric-api): Qualify the Looking Glass service on the lab - #1142
Conversation
The Looking Glass service is consumed by a customer-facing service outside this repository, so its qualification needs a consumer that uses the published API the way that service will, not the gateway's own packages. examples/looking-glass-client is that consumer: a separate module whose path lies outside go.datum.net/galactic, so the compiler refuses any import of Galactic's internal packages, and which imports only api/fabric/service/v1 and api/fabric/v1. It authenticates with a service certificate, pins the cell gateway's identity, and runs single diagnostics, workload specs (concurrency, bursts, per-project rates, slow or cancelling consumers) and arbitrary unary calls, printing one JSON record per attempt with its timing, terminal event and refusal details. It checks every stream it reads against the contract's event grammar and coverage rules. task test:unit now also builds, vets and tests the module, and fails if it imports any Galactic package beyond the published service API. task lint lints it. Related to #1048
The Looking Glass backend's acceptance requires executed evidence that its authorization, coverage, limits, cancellation and failover hold on real fabric routers, measured against thresholds fixed in advance, and that diagnostic load does not affect routing. Unit tests run the gateway against fake nodes and cannot show that. tests/lgqualify drives the service on dfw's fabric routers through the independent client in examples/looking-glass-client, with fault injection. Its scenarios cover service authorization and capability compatibility; router coverage when routers are selected, omitted or failing; the public destination policy; the operator and FabricQuery interfaces with the service on; cell, project and destination bursts and sustained fairness; per-destination packet emission across every caller; cancellation, deadlines, the service switch and a blackholed node; leader crash, pause and API-server isolation; BGP convergence with and without diagnostic load; gateway and sidecar resources; the absence of targets and projects from routine logs and metric labels; and, last, that no injected fault outlives the run. Each threshold comes from the service contract and is fixed in thresholds_test.go. Each scenario writes its observations and verdict as JSON, and a skipped check fails rather than passes. The lab gains what the run needs: - deploy:looking-glass enables the service on dfw in the documented order, through overlays in resources/looking-glass, and issues the suite's service certificates; undeploy:looking-glass reverses it. - tr4 announces a global IPv6 prefix for the off-fabric host, since the public probe policy refuses documentation space, and a sink prefix it drops silently. - build:fabric-api stamps the commit into the image, which the gateway reports as its contract revision. verify:looking-glass runs the suite. It restarts and pauses the gateway leader and resets one BGP session 80 times, so it is not part of verify. Related to #1048
f7fea78 to
fb6ac83
Compare
Operators need to know how to qualify a Looking Glass build before switching the service on in a cell, what each scenario proves, and what a lab pass does not establish. docs/fabric-api/qualification.md describes the procedure: what the suite exercises, mapped to the contract's acceptance rows, how its thresholds are fixed, how to run and roll back on the lab, where the evidence goes, and the lab's limits. The fabric-api index, the architecture guide's test section and the operations guide's verification section link it. The archive records the first passing run on the lab, with its measurements by acceptance row, the defects earlier runs found, and what the run does not establish, including the target platform. Related to #1048
fb6ac83 to
53d1a01
Compare
|
Two independent reviews ran on this PR, then a second pass on the fixed head. The first pass held on an overstated title, a commit subject over 72 characters, a stale script comment and an output path the lab run never used. All four are fixed, and a dry run resolves the path to the repository's ignored bin directory. The second pass split, holding on the archive record's list of differences from the tested commit, which now names the script comment too. The lab run recorded in the archive used commit 980cc91c. The published commits differ from it only in documentation, one gateway unit test, the lab task's output directory and a script comment, so the suite was not rerun on this head. A trial merge with current main, after #1140, has no conflicts. CI is green on the current head. Merge the stack from this PR with the merge-async endpoint once both layers are approved, since GitHub does not allow auto-merge on stacked pull requests. This is related to #1048, which stays open until the suite runs on a staging cell. |
ecv
left a comment
There was a problem hiding this comment.
Approve.
Lab-only change: the client, suite, containerlab config and docs ship nothing to production, and the pass criteria are fixed thresholds that fail the run rather than pass vacuously. It is stacked on #1141 and must merge after it.
- Stacked: the base is #1141's head (
fix/lg-qualify-findings), so this diff is only the qualification work. It needs #1141's refusal policy, since the lab probe sink moved to global2c0f:40::/64and11.1.42.0/24because documentation space is now refused, and its client-auth certs for direct operator probes. Onlydocs/fabric-api/operations.mdoverlaps (one added paragraph, no conflict). Merge #1141 first and let GitHub retarget this PR. - Nothing ships: no
.github/,config/orcontainers/file changes. Lab overlays live underdeploy/containerlab/resources/looking-glass/and only reference existingconfig/fabric-apicomponents. The client is its own module underexamples/, andtest:looking-glass-clientfails if it imports Galactic packages beyond the published service API. - Thresholds are consts in
thresholds_test.go, fixed from the design doc. The scenarios fail on a miss, and the routing-impact scenario treats a control p95 whose bootstrap half-width is over 10% as a failure, not a pass.destinationsasserts refusal with the exact reason for private, documentation, operator-allowed loopbacks, multicast and hostnames, andTestQualifycallst.Fatalif the client or binaries are missing. - Without
LG_QUALIFY_KUBECONFIGthe suite skips, so a barego test ./...in CI passes without running it. That fits a 45-minute lab run, but a green check proves nothing about the Looking Glass. The archive benchmark file is the evidence of an actual run. destinationschecks the refusals but not that they cost no tokens, slots or packets, which is the pre-charge property #1141 adds.
Follow-ups:
- Assert in
destinationsthat the project and cell budgets are unchanged after a refused probe. - Add
--deny-prefixesto the dfw gateway overlay so the lab exercises the gateway-side platform refusal and not only the special-use ranges.
Summary
The Looking Glass backend needs executed evidence on real fabric routers, against thresholds fixed in advance, before any cell turns the service on, and unit tests against fake nodes cannot supply it.
This adds an independent client, built only from the published service API, and a qualification suite that drives the service through it on the containerlab lab with faults, covering every row of the contract's acceptance matrix, including routing impact and cleanup.
A full run on the lab passed all 21 scenarios, with loaded BGP convergence at 1.00 times the control, and the qualification guide and an archived record describe how to repeat it and what it measured.
A lab pass does not qualify a staging or production cell, so the issue stays open until the suite runs there.
Stack
Second of two stacked PRs, on top of #1141, which fixes the four defects earlier runs of this suite found. Nothing here changes a shipped binary or manifest. The lab image build now records its commit, and rolling back means reverting this.
Test plan
Related to #1048