Repository navigation
Conversation
tnunamak
force-pushed
the
feat/collection-profile-declaration
branch
2 times, most recently
from
November 6, 2026 12:34
a8093a1 to
68de4cc
Compare
…d source identity A 0.2.0 manifest declares what it needs (bindings, capabilities, human interaction kinds) and what source it speaks for (source declaration). The runtime had no single place that decided whether it could honor that declaration, so a connector could be admitted and then fail late. The runtime now: - keeps one support declaration, `REFERENCE_COLLECTION_PROFILE_SUPPORT`, used by the registration gate and the run path, and serves it to the owner session at `GET /_ref/collection-profile` before any run; - parses `runtime_requirements.bindings` with the pinned grammar and schema (browser, desktop_session, filesystem, network and namespaced names). A required binding it cannot supply is refused before spawn, and START.bindings lists only what it supplies; - refuses a required capability it does not implement (BLOB is no longer tolerated), and stops advertising DETAIL_COVERAGE_KEY_SETS in START; - requires an INTERACTION.kind to be declared in `capabilities.human_interaction` instead of an interactive binding; - refuses a run before spawn with `failure_reason: protocol_capability_unsupported` when the declaration or a required capability cannot be honored; - keeps a registered connector's source identity once retained, and requires a first registration of an unmapped key to carry an explicit source_declaration. The installer verifies the source-declaration sidecar digest. Historical first-party mappings are pinned and checked by the registry generator. Existing tests now declare their fixture source and human interaction kinds explicitly. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
… activation The installer publishes a verified digest root with a rename and then writes the active record. A crash between the two left a root that verified against the catalog entry but had no record, and every later install of that digest threw "root already exists without a matching active record". The owner could not recover without deleting the root by hand. The install now activates and registers a root that re-verifies against the entry, the same way it reuses a root that has an active record. A root that does not verify is still removed. A symlinked root is still refused. A named crash point after the rename lets the kill-and-restart test reach that window. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Three test files register a manifest with no `source_declaration` and a key that has no reviewed source mapping. The registration gate in this branch refuses that, so the files failed at setup, not at the behavior they test: stream-evidence, the startup semantic backfill crash test and the search-index repair test on PostgreSQL. Each fixture now declares its source with the shared helper, the same way the other fixtures do. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
… record instead of adopting it The previous commit on this branch adopted a published digest root that had no active record: it hashed the files found there into a new record and activated it. That let the bytes on disk vouch for themselves. A root altered between the crash and the retry (the executable, the provenance file or the source-declaration sidecar) became the verified install, and an altered sidecar could be recorded as the pinned declaration. With no record there is no retained verification, so nothing in the root can be trusted. The install now removes the leftover root and installs again from the verified artifact, which is what it already did for a root that failed its checks. The crash window still heals without manual cleanup, and the crash point after the rename stays so the kill-and-restart test reaches it. Tests: the happy-path test is renamed to say a retry reinstalls. Two new tests alter the root after the crash (the executable alone; the executable, provenance, an added sidecar and an extra file) and check that the retry installs the artifact's bytes and nothing the crash left survives. Both failed against the adopting code. Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…Collection Profile An installed profile was read with JSON.parse, which keeps the last value of a repeated key. A bindings map that declared `network` twice reached registration as one binding with the second value. The HTTP and local source paths already reject a repeat (parseManifestJsonText), so one declaration was accepted or refused depending on how it arrived. Both reads of the profile in the installer, at install time and when an active record is verified, now use parseManifestJsonText on the text as written. The error says only that the profile repeats an object key: the text is not trusted and the error can reach the owner, so it carries no key or path. Tests: a repeat at the bindings level, a repeat nested in one binding, and an escaped spelling of the same key are each refused at install with nothing active, nothing registered and the root removed. A record whose profile bytes repeat a key under their own recorded hash is not runnable. All four failed before. Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak
force-pushed
the
feat/collection-profile-declaration
branch
from
November 6, 2026 17:19
e6affb6 to
a4fb70d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base: #630. Merge that first, or retarget this PR to
mainafter it lands.What changes
Before this PR the runtime had no single place that decided whether it could honor a manifest's Collection Profile declaration. A connector could be admitted and then fail late, and a connector's source identity was inferred from its
connector_idormanifest_uri.REFERENCE_COLLECTION_PROFILE_SUPPORT(runtime/collection-profile-support.ts) lists the protocol versions and capabilities the runtime implements (OBSERVATION,STREAM_EVIDENCE) and disclosesbinding_authorityas outstanding. The registration gate and the run path both use it. The owner session reads it atGET /_ref/collection-profilebefore any run.runtime/binding-declarations.tsparsesruntime_requirements.bindingswith the pinned grammar and schema (vendored byte copies,connector-binding-grammar.mjs,connector-manifest.schema.json). A duplicate object key is refused atPOST /connectorsand when stored manifests are read. A required binding the runtime cannot supply is refused before spawn. START lists only the bindings it supplies.BLOBis no longer tolerated, so a manifest that declares it is refused. START no longer advertisesDETAIL_COVERAGE_KEY_SETS.INTERACTION.kindmust appear incapabilities.human_interaction. The oldinteractivebinding is gone.failure_reason: protocol_capability_unsupportedinstead of a plainError. This covers an unsupported required capability, a malformedprotocol_capabilitieslist, and an inadmissible profile declaration.server/auth.ts,connector-legacy-source-map.ts). A first registration of a key with no reviewed mapping needs an explicitsource_declaration. The installer verifies the digest of thesource-declaration.jsonsidecar. The historical first-party mappings are pinned and checked byscripts/legacy-source-binding-validation.tsin the registry generator.Defect fixed in this PR
cbf166de52, afterb01869a5ee). The installer renames a verified root into place and then writes the active record. A crash between the two left a root with no record, and every later install of that digest threw "root already exists without a matching active record". The first fix adopted such a root by hashing the files found there into a new record. That let the bytes on disk vouch for themselves, so a root altered after the crash became the verified install. The install now removes a root that has no active record and installs again from the verified artifact, as it already did for a root that failed its checks. The crash window heals without manual cleanup and nothing the crash left can become active. The crash pointconnector-install:after-publishstays. Tests inconnector-install-registration-crash.test.ts: a retry reinstalls; a root altered after the crash (executable only; executable, provenance, a sidecar and an extra file) is replaced by the artifact's bytes. The two alteration tests failed against the adopting code. The same pattern exists onmain, where a retained root of another digest is reused when the active record names a different digest (reuseExistingInstallwith no record for that root); that path is not changed here.Repeated object keys in an installed profile (
e6affb6d1e)The installer read an installed Collection Profile with
JSON.parse, which keeps the last value of a repeated key. A bindings map that declarednetworktwice reached registration as one binding. The HTTP and local-source paths already reject a repeat, so one declaration was accepted or refused depending on how it arrived. Both profile reads inconnector-install/index.ts(at install and when an active record is verified) now useparseManifestJsonTexton the text as written. The error says only that the profile repeats an object key, with no key or path. Tests inconnector-install.test.ts: a repeat at the bindings level, a repeat inside one binding and an escaped spelling are each refused at install with nothing active, nothing registered and the root removed; a record whose profile bytes repeat a key under their own recorded hash is not runnable. All four failed before.Fixture follow-up (
68de4ccb33)Three fixtures registered a manifest with no source declaration, which this PR's gate refuses:
stream-evidence,semantic-startup-backfill-catches-crash-abandoned-recordandsearch-index-repair-correctness.postgres. They now declare the source withwithFixtureSourceDeclaration. Memory: 16 and 43 pass. PostgreSQL: 6 pass.Checked and left alone
reconcilePolyfillManifests, wired inserver/index.ts) already registers the manifest of an installed connector the registry lacks.connector-install-registration-crash.test.tskills the child atconnector-install:after-activateand asserts the registry is restored on the next start, for a first install and an update. Both pass here.assertStoredFileHashes. It hashes the manifest, entrypoint and provenance, which are every file the pinned installer writes. Sibling code files appear only if a later installer version writes them, so this is not a defect today.publishConnectorActivationandcompleteConnectorActivation. Neither has a production caller, here or onmain; onlyremoveConnectorActivationis called. This PR extends the module to carry the source identity, and tests call it. Wiring or deleting it changes the install flow, so it is not done here. Decision needed from a maintainer.Differences from the working source
collection-profile.test.tshint cases,scheduler.test.tsretry cases).connector-install/index.tsdoes not carry theinspecthook from the browser-admission work.Upgrade and rollback
A manifest or connector written for 0.1 can stop working under this PR. A manifest that declares an
interactivebinding in any form is refused before spawn, because the canonical schema has no such binding (runtime/index.ts,normalizeBindingDeclarations). A connector that emits an INTERACTION whosekindis not incapabilities.human_interactionfails the run (runtime/index.ts, "Connector emitted INTERACTION.kind not declared in capabilities.human_interaction"). The old success case is replaced incollection-profile.test.ts. This PR adds no compatibility mode for either case.The support document and the typed refusal reason are additive. An older console shows the refusal text as it does today. There is no down-migration for the retained source identity: roll back by restoring the backup taken before the upgrade.
Verification
Run from
reference-implementation/. Pass rule: exit 0 and notest:fail.Targeted runs on the final tree, one file each (memory unless noted):
collection-profile: 149 pass.collection-profile-020: 21 pass, and 21 pass on PostgreSQL.scheduler: 43.event-spine: 53.run-interaction-stream-routes: 62.run-interaction-control: 12.ri-zero-connector-knowledge-conformance: 116.owner-connection-intent: 27.connector-installandconnector-install-registration-crash: 64 together.postgres-template-eligibility-inventory: 3. Also green:runtime-stream-collection-failed-commit,scheduler-static-secret-injection,checkpoint-dependency-profile-conformance,auto-enroll-eligible-schedules-integration,browser-enrollment-shell-route,owner-connector-templates,composed-origin, bothconnector-state-scheduler-conformancefiles,local-source.test.ts, and the 21 files of the focused list.connector-install-registry-integration,connector-install-sql-persistence,connector-source-identity-boundary(the PostgreSQL cases run; 0 fail).tsc --noEmitpassed before the last test edits.a8093a1b12, after the rebase onto the then-current feat(runtime): accept OBSERVATION messages and keep a per-run diagnostic record #630 head (the recovery-hint gate moved to feat(runtime): stop acting on connector claims about recovery and coverage #636), memory unless noted:tsc --noEmitexit 0;collection-profile149,collection-profile-02024 (24 on PostgreSQL),collection-profile-binding-declarations27,collection-profile-owner-journey1,connector-install-source-declaration20,connector-install-registration-crash9,connector-install55,connector-registry-manifest-derivation10,runtime-observations48,scheduler43,detail-coverage-shortfall-severity28,stream-evidence16,runtime-stream-collection-failed-commit14,checkpoint-dependency-profile-conformance8,manifest-runtime-requirements5,event-spine53,cli141,local-source8. PostgreSQL:connector-observation-store22,connector-install-postgres17,connector-install-sql-persistence2,checkpoint-dependency-profile-conformance-postgres5,connector-source-identity-boundary85. 0 fail in every file.e6affb6d1e), 0 fail:tsc --noEmitexit 0; memoryconnector-install59,connector-install-registration-crash11,connector-install-source-declaration20,connector-install-registry-integration1 (1 skipped),connector-install-sqlite1,connector-source-identity-boundary43 (42 skipped),collection-profile-binding-declarations27,connector-registry-manifest-derivation10,owner-connector-templates16,local-source8,collection-profile-02026,collection-profile149; PostgreSQLconnector-install-postgres17,connector-install-sql-persistence2,connector-install-registry-integration2,connector-source-identity-boundary85.68de4ccb33):tsc --noEmitexit 0; memorycollection-profile-02028,collection-profile-binding-declarations27,connector-install-source-declaration20; PostgreSQLcollection-profile-02028 andconnector-observation-store22. 0 fail.Still owed: a full-suite run on memory and on PostgreSQL for the final head. A full memory run on an earlier head ran 11,273 tests before it was stopped for a host-load freeze; every failure it showed has been fixed and re-run file by file as listed above, but the tail of that run did not finish. The one known failure that also fails on
mainisconnector-failure-diagnostics"stderr-bearing exit before DONE" (a Nodemodule.register()deprecation line in the child's stderr).Assisted-by: AI