Artifacts Lint: Spectral step loses finding attribution and under-reports the real findings

[hdamker]

Problem description

The Spectral step of the Artifacts Lint workflow (#673) lints all four templates in a single Spectral invocation. Spectral loses the source file of most findings when several input documents share $ref targets (stoplightio/spectral#2640), so the step reports the same result three different ways and spends most of its annotation budget on findings it cannot attribute.

Run 30385534061 (job 90363770115), on the head of #678:

Surface	Count
pretty summary on stdout	✖ 9 problems (0 errors, 9 warnings, 0 infos, 0 hints)
::warning lines in the job log	152
annotations on the check run	20 — 10 from the Spectral step, 10 from the gplint step

GitHub renders at most 10 annotations per level per step. 143 of the 152 findings carry no source, so they annotate . at line 1: three of the Spectral step's ten slots go to those unattributed annotations, and the two real findings in api-templates/sample-service.yaml (lines 278 and 293) get no annotation at all. A contributor reading one step sees three numbers — 9, 20, 152 — for one result.

Reproduced against main with the pinned toolchain (Spectral 6.16.2, .spectral-r4.yaml); the counts are identical to the run above, so nothing here is specific to #678:

• one invocation over all four templates → 152 results, 9 with a source, 143 with none (range 0:0)
• one invocation per template → 26 results, none without a source, 15 unique (file, line, rule)

Template linted alone	Findings	Without source
api-templates/sample-implicit-events.yaml	7	0
api-templates/sample-service-subscriptions.yaml	10	0
api-templates/sample-service.yaml	5	0
notification-templates/sample-notification.yaml	4	0

The behaviour is deterministic and independent of argument order, and adding documents shrinks the attributed set (two templates → 12 sourced, four → 9).

The pretty summary is therefore not a reference surface — its 9 findings are a lossy subset of the real 15, and six genuine findings are invisible in the current output entirely:

• common/CAMARA_common.yaml:57 — the pagination Link header
• common/CAMARA_event_common.yaml lines 111, 173, 393, 416, 441

Expected behavior

Lint each template in its own invocation and deduplicate the results on (file, line, rule). CAMARA Validation already does exactly this for the same reason — run_spectral_engine in the tooling repository invokes Spectral per file and cites the same Spectral issue — so the workflow converges with the validation toolchain instead of diverging from it. The pretty output, the annotations and the reported counts then all describe the same 15 findings.

Ten of those 15 are the documented common-library baseline, already recorded in the suppress_schema_paths allowlist of rule S-313 in the tooling rule metadata. Bare Spectral does not read that metadata, so the step should apply the allowlist itself: collapse the findings it covers into a single notice annotation per source file, carrying that file's count and pointing at the documented list, and annotate everything else individually at its native Spectral severity.

The cap is 10 annotations per level per step, so collapsing the baseline to notice takes it off the warning budget entirely instead of merely shrinking its share of it — all 10 warning slots stay available for findings a change actually introduces, against 10 of 10 consumed by the baseline today. Keying the collapse on the allowlisted locations rather than on the rule code preserves the property that allowlist is built for: a newly added unconstrained string in a common schema still surfaces on its own and can be judged on its own merits.

On the current baseline that is 2 notice annotations (common/CAMARA_common.yaml, common/CAMARA_event_common.yaml) and 5 individual warning annotations.

Alternative solution

• Filter out the source-less lines. Makes the annotations honest but cements the lossy 9-finding view, so the six findings listed above stay invisible.
• Lint a bundled specification instead. Attributes findings to a generated file rather than to artifacts/common/**; the tooling documentation describes bundles as a preview and not a means of locating the source of a validation problem.
• Drop -f github-actions. Gives up the annotations Add Artifacts Lint workflow (yamllint, Spectral, gplint) #673 added and leaves the under-reporting unfixed.

Additional context

#676 documents the expected findings of the artifacts. Its recorded baseline of 9 Spectral owasp:api4:2023-string-restricted findings is the buggy multi-document view; the real baseline is 15 — 11 in the common files and 4 name fields in the API templates. The classification in #676 is unaffected, only the count.