feat(release): the compiler writes envelope 3, held to the frozen conformance yardstick

docs/en/RELEASE-COMPILER-ENVELOPE3.md

@@ -0,0 +1,52 @@
+# The compiler writes envelope 3
+
+`python tools/compiler.py release` writes a `PrimitiveRelease` record of **envelope 3** (docs/en/RELEASE-RECORD-CONTRACT.md) and nothing else. Envelope 1 and 2 records are not written any more, and are not touched: `verify-release` still reads them exactly as before, and the real records `release/cic-primitives-v0.1.5.yaml` (envelope 1) and `release/cic-primitives-v0.2.0.yaml` (envelope 2) are test cases. No record is rewritten, re-signed or migrated.
+
+## What changes in a release
+
+1. `provenance.source_commit` is mandatory: `git rev-parse HEAD`, 40 lowercase hexadecimal digits. If git cannot say, the compiler stops; it no longer records `null`.
+2. The producer declares the files it publishes in `project.yaml`:
+
+   ```yaml
+   compiler_settings:
+     release_artifacts:
+       paths:            # explicit, relative to the repository root; no glob, no include, no exclude
+         - LICENSE.md
+         - schemas/atomic/shape.yaml
+   ```
+
+   The declaration is read from **the source commit** (`project.yaml` as that commit holds it), and the paths are resolved against **the git tree of that commit**: not the working directory, not a checkout with its line-end filters, not a consumer's pin manifest. A path that is a directory, a symbolic link or a submodule is refused, as is an absolute path, a `..` or `.` segment, a control character, a repeated path, a missing path, an empty list, a key other than `paths`. The compiler stops with the reason (`declaration_missing`, `declaration_empty`, `declaration_unknown_key`, `declaration_path_form`, `declaration_path_duplicate`, `declared_path_absent`, `declared_path_not_regular`) **before** it asks Vault for anything.
+3. `artifacts` is written: for each declared path `sha256:` and the hex SHA-256 of the blob bytes at the source commit, sorted by the UTF-8 bytes of the path, and `manifest_digest` over the sha256sum-format lines. `specs[].source_path` must be unique.
+4. `build_hash`, `release.sign` and (outside the compiler) `cic_countersign` are the existing mechanism: `build_hash` covers the whole bundle except `release.sign`, `release.build_hash` and `cic_countersign`, so `artifacts` and `source_commit` are covered without any change to it.
+
+A repository whose `project.yaml` has no `release_artifacts` cannot cut a release: it stops with `declaration_missing`. Which files a repository publishes is the producer's decision and is made in `project.yaml`, in a change of its own.
+
+## The rest of the record is the source commit's too
+
+`artifacts` is read from the git tree. The rest of the record is not made from the tree: `specs` (their content and `meta_hash`) come from the schema files, the three legacy provenance digests from `dependency.yaml` and the two grammar files, the issuer and the validity from `project.yaml` and `commitment.yaml`. `validate_release_prerequisites` asks `git status --porcelain`, and a clean status does **not** make those the commit's bytes: an ignored file, a file marked assume-unchanged or skip-worktree, a line-end filter or a sparse checkout all leave it clean (each is a test, with the premise that the status is clean asserted first).
+
+So `run_release` compares, before it asks Vault for anything, every one of those inputs with the blob of the source commit, and the schema files it found with the selection made on the commit's tree. A difference (`worktree_differs_from_source_commit`, with the file and what differs) stops the release: a record made of two states of the source would contradict itself without anyone having lied.
+
+An input that exists **neither** in the working directory nor in the source commit is not consistency either: `project.yaml`, `commitment.yaml` and the two grammar files are required inputs of the release and the check names them itself (`dependency.yaml` is the one optional input: without it the legacy digest is `null`, as before). Each is a test end to end with the real commitment loader.
+
+**Every published schema is listed.** Every `specs[].source_path` the record would publish must be among `artifacts.files` (`spec_not_in_artifacts`, naming the files, before any Vault call). A published schema cannot be missing from the artifact manifest. This is a producer-side refusal of the compiler; it is not a rule of the record contract (a record-level check would be a revision of the contract and of the frozen yardstick, and is not made here).
+
+What this does not cover: the validation gates (`run_validation`, `run_primitive_validation`, the grammar gate) read the working directory too. Their verdict can concern a file that is not part of the record (an ignored example, for instance) and is not about the commit for such files; it does not enter the record. A release should be cut from a fresh clone of the commit. The boundary is pinned by two characterization tests (not wishes): a composition that the commit holds and the grammar rejects, left out of the working directory with `git status` clean, passes the gate and the release is cut; and an ignored bad example that the commit does not hold stops a release of a good commit. If the boundary is ever closed, those tests change on purpose.
+
+## How the output is held to the contract
+
+The compiler does not say that its own output is right.
+
+* The record it writes is read back by the reference verifier (`tools/release_record_conformance.py`), which does not import the compiler's new functions (and the compiler does not import it): `record`, `blobs` and `crypto` layers, and the `artifacts` block is compared with values recomputed outside this code base (`tests/fixtures/envelope3`).
+* The 32 signature coverage vectors of the contract run on the record the compiler wrote, not only on the golden record; the structural vectors too.
+* The declaration vectors run on the compiler's own functions, and on the reference's, and the two must agree.
+* The yardstick of PR #25 (the reference verifier, the golden fixtures, the vectors, the planted-defect proof) is frozen at its merge commit: `tests/test_conformance_frozen.py` fails if any of it changes together with the compiler.
+* 82 planted defects in `tools/compiler.py` (`tests/e3_compiler_mutants.py`): 72 have to turn one of the tests that target them red, 10 cannot change behaviour, are listed with the reason and have to survive.
+
+## `verify-release`
+
+Accepts envelope 3 (build hash, signature, counter-signature, as for envelope 2) and says what it leaves to the conformance check: the artifact list is **not** checked there. The two are separate: `verify-release` is the cryptographic verification, `tools/release_record_conformance.py` is the check against the envelope 3 contract. A release policy may require both; neither imports the other.
+
+## The counter-signature service
+
+Read-only audit of `cic-countersign` (`internal/countersign/handler.go`, `devel` 8ce2427): the service takes `build_hash` as an opaque, already computed digest ("trusted as given, never recomputed") and verifies the author's signature over it. The submitted `document` is carried through for the audit log only (`kind` and `version` are logged) and is not hashed or canonicalized. It reads nothing of the envelope or of `artifacts`. Nothing there depends on this change; the service was not modified.

docs/hu/RELEASE-COMPILER-ENVELOPE3.md

@@ -0,0 +1,14 @@
+# A compiler envelope 3-at ír
+
+A `python tools/compiler.py release` kizárólag **envelope 3** `PrimitiveRelease` rekordot ír (docs/en/RELEASE-RECORD-CONTRACT.md). Az envelope 1 és 2 rekordot nem ír többé, a meglévőket nem érinti: a `verify-release` változatlanul olvassa őket, a valódi `cic-primitives-v0.1.5.yaml` (envelope 1) és `cic-primitives-v0.2.0.yaml` (envelope 2) tesztesetek. Rekord átírása, újraaláírása vagy migrációja nincs.
+
+* **`source_commit` kötelező** (`git rev-parse HEAD`, 40 kisbetűs hex); ha a git nem tud felelni, a compiler megáll, nem ír `null`-t.
+* **A termelő deklarálja a közzétett fájlokat** a `project.yaml`-ban (`compiler_settings.release_artifacts.paths`, explicit útvonalak, glob/`include`/`exclude` nélkül). A deklarációt a **forráscommit**ból olvassa, az útvonalakat a commit **Git-fájából** oldja fel; nem a munkakönyvtárból, nem szűrt checkoutból, nem fogyasztói pin-manifestből. Könyvtár, link, submodule, abszolút útvonal, `..`/`.`, vezérlőkarakter, duplikátum, hiányzó útvonal, üres lista, ismeretlen kulcs: leáll az okkal, **még a Vault hívása előtt**.
+* **`artifacts`:** útvonalanként `sha256:` + a blob bájtjainak SHA-256-ja, az útvonal UTF-8 bájtjai szerint rendezve, plusz `manifest_digest`. A `specs[].source_path` egyedi.
+* **Aláírás:** a meglévő mechanizmus (`build_hash` a teljes csomagra, `release.sign`, külső `cic_countersign`); az `artifacts` és a `source_commit` ezért lefedett.
+* **Nem a compiler igazolja magát:** a kimenetet a referencia-verifier olvassa vissza (nem importálják egymást), a 32 lefedettségi vektor a compiler rekordján is lefut, a deklarációs vektorok a compiler saját függvényén, és a #25 mércéje befagyasztott (`tests/test_conformance_frozen.py`). 82 planted defect a `tools/compiler.py`-ban.
+* **A rekord többi része is a forráscommité.** Az `artifacts` a git-fából jön; a `specs` (tartalom, `meta_hash`), a három régi provenance-digest, a kiadó és az érvényesség viszont a munkakönyvtárból (séma-fájlok, `dependency.yaml`, két grammar-fájl, `project.yaml`, `commitment.yaml`). A tiszta `git status` ezt **nem** garantálja (ignorált fájl, assume-unchanged, skip-worktree, sorvég-szűrő, sparse checkout mind tiszta státuszt hagy). A `run_release` ezért a Vault előtt minden ilyen bemenetet a forráscommit blobjához hasonlít, és a talált séma-fájlokat a commit fáján végzett kiválasztáshoz; eltérés: `worktree_differs_from_source_commit`, leáll. Nem fedi: a validációs kapuk (séma-, primitive-, grammar-kapu) is a munkakönyvtárat olvassák; ítéletük olyan fájlra is szólhat, ami nem része a rekordnak. Release-t friss klónból érdemes vágni.
+* **Kötelező bemenet és a közzétett sémák:** ami sem a munkakönyvtárban, sem a commitban nincs, az nem konzisztencia: a `project.yaml`, a `commitment.yaml` és a két grammar-fájl kötelező bemenet, a guard maga nevezi meg (a `dependency.yaml` az egyetlen opcionális: nélküle a régi digest `null`). Minden `specs[].source_path` benne kell legyen az `artifacts.files`-ban (`spec_not_in_artifacts`, a Vault előtt); ez termelői oldali elutasítás, nem rekord-szerződési szabály. A fennmaradó határt két jellemző teszt rögzíti: a validációs kapuk a munkakönyvtárat ítélik (a commitban lévő, de a munkakönyvtárból kihagyott rossz példát a kapu nem látja, az ignorált rossz példa viszont megállítja a jó commit release-ét).
+* **`verify-release`:** elfogadja az envelope 3-at, és kimondja, hogy az artifact-lista ellenőrzése nem az ő dolga (`release_record_conformance.py`); a kettő különálló, egyik sem importálja a másikat.
+* **Ellenjegyző szolgáltatás (csak olvasó vizsgálat):** a `cic-countersign` a `build_hash`-t átlátszatlan, kész digestként kapja és nem számolja újra; a `document` csak naplóhoz; az envelope-ból és az `artifacts`-ból semmit nem olvas. Nem módosítottam.
+* **Deklaráció nélküli repó nem adhat ki release-t** (`declaration_missing`); a közzétett fájlok listája a termelő döntése, külön változásban.

project.schema.yaml

@@ -99,6 +99,19 @@ properties:
       vault_cert_path:
         type: string
         description: "Vault KV v2 path to the signing certificate. Format: 'mount/secret-name/key'. Overridden by VAULT_CERT env var."
+      release_artifacts:
+        type: object
+        description: "The files a release publishes (docs/en/RELEASE-RECORD-CONTRACT.md, section 6): explicit paths relative to the repository root, resolved only against the git tree of the source commit. No globs, no include or exclude. Read from the source commit, not from the working directory."
+        additionalProperties: false
+        required:
+          - paths
+        properties:
+          paths:
+            type: array
+            minItems: 1
+            items:
+              type: string
+              minLength: 1
 
   x-cic:
     type: object

release.schema.yaml

@@ -123,9 +123,12 @@ properties:
         enum:
         - 1
         - 2
+        - 3
         description: "Which payload build_hash covers. 1 = the four members {createdBy, releasedBy, specs,\
           \ validity} \u2014 legacy, leaves kind/version/timestamp unsigned. 2 = the whole bundle except\
-          \ release.sign, release.build_hash and cic_countersign. Absent means 1."
+          \ release.sign, release.build_hash and cic_countersign. 3 = the same payload rule as 2, and\
+          \ the bundle carries `artifacts` and a mandatory `provenance.source_commit` (docs/en/RELEASE-RECORD-CONTRACT.md).\
+          \ Absent means 1."
   cic_countersign:
     type: object
     description: External CIC counter-signature over release.build_hash. Applied by CICSourceCA after
@@ -184,3 +187,57 @@ properties:
         type:
         - string
         - 'null'
+  artifacts:
+    type: object
+    description: 'Envelope 3: a digest of every released file (docs/en/RELEASE-RECORD-CONTRACT.md, sections 6 to 8). sha256: and 64
+      lowercase hexadecimal digits of the git blob bytes at provenance.source_commit; files sorted by the UTF-8 bytes of the path, unique;
+      manifest_digest over the sha256sum-format lines. This schema checks the shape only; tools/release_record_conformance.py is the reference check.'
+    required:
+    - manifest_digest
+    - files
+    properties:
+      manifest_digest:
+        type: string
+        pattern: ^sha256:[0-9a-f]{64}$
+      files:
+        type: array
+        minItems: 1
+        items:
+          type: object
+          required:
+          - path
+          - digest
+          properties:
+            path:
+              type: string
+              minLength: 1
+            digest:
+              type: string
+              pattern: ^sha256:[0-9a-f]{64}$
+allOf:
+- description: An envelope 3 record carries artifacts and a source_commit that is 40 lowercase hexadecimal digits (never null).
+  if:
+    required:
+    - release
+    properties:
+      release:
+        required:
+        - envelope
+        properties:
+          envelope:
+            const: 3
+  then:
+    required:
+    - provenance
+    - artifacts
+    properties:
+      provenance:
+        required:
+        - envelope
+        - source_commit
+        properties:
+          envelope:
+            const: 3
+          source_commit:
+            type: string
+            pattern: ^[0-9a-f]{40}$