diff --git a/docs/decisions/0008-the-experiment-record.md b/docs/decisions/0008-the-experiment-record.md index 579b710..901f91d 100644 --- a/docs/decisions/0008-the-experiment-record.md +++ b/docs/decisions/0008-the-experiment-record.md @@ -1,8 +1,9 @@ # 0008. What an experiment record looks like -Superseded by 0014, which fixes what a slug may be, and by 0015, which adds the -field naming what an experiment needs beyond the runner. Everything below stands -as it was written. +Superseded by 0014, which fixes what a slug may be, by 0015, which adds the +field naming what an experiment needs beyond the runner, and by 0016, which adds +the field naming the commit a measurement was produced at. Everything below +stands as it was written. ## What was decided diff --git a/docs/decisions/0016-an-answer-names-the-commit-it-measured.md b/docs/decisions/0016-an-answer-names-the-commit-it-measured.md new file mode 100644 index 0000000..799f322 --- /dev/null +++ b/docs/decisions/0016-an-answer-names-the-commit-it-measured.md @@ -0,0 +1,113 @@ +# 0016. An answer names the commit its measurement was produced at + +## What was decided + +This record adds a field to the experiment record format record `0008` fixes, +and it supersedes that record for that one thing. Everything `0008` says stays +as it was written, and how the format grows at all is record `0013`, which this +answers to rather than restates. + +An answer that quotes a measurement carries `Measurement-Commit` in the header, +whose value is the object name of the commit the measurement was produced at, +written in full. + + Measurement-Commit: e5067b1d0f4a2c8b6e3a9d7c1b5f8a2e4d6c0b93 + +The field is optional, because record `0013` makes every field added after it +optional and an absence is never refused. What is refused is a value that is not +the shape of an object name, which is the whole of what a checkout can be asked. + +The full name rather than an abbreviation. Record `0004` already writes the +commit that removed an experiment's code with the full hash, for the reason an +abbreviation stops being unique as a repository grows, and a measurement is the +same shape one step earlier. A reader who cannot resolve the name they were +given is in the position this field exists to remove them from. + +Why a field rather than a sentence in the prose. An answer already names the +command, the platform, the architecture and the toolchain, and none of that is +read by anything. What is missing is the version of the code the command ran +against, and record `0004` lets that code be removed entirely afterwards. When +it is, the answer keeps its numbers and loses the thing that produced them, and +it does so silently, because the record still reads as complete. A field is +where the runner already looks. + +What this does not buy, written here because a green run will otherwise be read +as more than it is. The runner reads a checkout and opens no connection, so +nothing here asks git whether the object named is in this repository, or whether +it is a commit, or whether the command in the answer was ever run against it. A +record naming forty hexadecimal characters that resolve to nothing passes. +What the refusal converts is the case where the value could not be resolved by +anybody, which is the only case a checkout can separate from the rest. + +And the absence stays unrefusable. A record quoting a measurement and carrying +no `Measurement-Commit` is legal, so the field makes the fact recordable and +never guaranteed. That cost is record `0013`'s, paid deliberately, and the +things that make a field usual without making it required are the template, the +review, and this record. + +## What it applies to + +Every `EXPERIMENT.md` under `experiments/`, from the commit this record lands +on, and the check that reads the field. + +It applies to what is present. A record already on the default branch is not +edited, not migrated and not marked, which is record `0013` and is why the one +record on the board when this landed was left exactly as it was. + +It applies to the template, which names the field in its prose rather than +carrying it in its header, for the reason `Answer-Written` is named the same +way: a template that ships a field filled in teaches every new record to declare +something it has no value for yet. + +It does not apply to the decision records in `docs/decisions/`. Those carry the +four sections record `0000` fixes and change by superseding. + +It does not decide anything about a measurement produced somewhere other than +this repository. An answer measuring another project's code has the same +problem and a different answer, and nothing here should be read as covering it. + +## What else was considered + +A convention in the answer prose, in the words the record already uses, with +nothing reading it. + +Nothing at all, on the argument that a reader who wants the exact code has git +and the record's own history. + +A field carrying the date of the measurement rather than the commit. + +A field required of every record whose answer quotes a command. + +Naming the commit with an abbreviated hash. + +## What each rejected option would have cost + +A convention costs exactly what it is: nothing reads it. The failure this is +about is a record that still reads as complete after the thing that produced its +numbers has gone, and a convention is another sentence in the file that already +read as complete. It also drifts, because two authors writing the same +convention in their own words produce two shapes, and the first check anybody +writes over it has to be right about both. + +Nothing costs the reader the work, and it charges the reader least able to do +it. Somebody who was there can find the commit from the date and the log. +Somebody who was not is the person this whole record is for, and the answer they +get is that the numbers were true of some version of a directory that record +`0004` may have removed. + +A date costs the precision that makes the field worth having. Several commits +land on one day here, and the answer already carries `Answer-Written`, so a date +would duplicate a field that exists and still not say which code ran. + +Requiring the field costs the format its ability to grow, which is record +`0013`'s whole argument. It would also require the checker to decide what counts +as an answer quoting a measurement, which is a judgement about prose that no +reading of a tree makes: an answer holding a number and an answer holding a +number that was measured look the same to everything in this repository. + +An abbreviation costs uniqueness at the moment somebody needs it. A short name +resolves today and stops resolving when the repository grows into a collision, +and the reader who meets that is the one who came back years later, which is the +case the field exists for. It also cannot be told from a typo: seven characters +that resolve to nothing and seven characters somebody mistyped are the same +string. diff --git a/docs/experiment-template.md b/docs/experiment-template.md index 998664d..cfea49b 100644 --- a/docs/experiment-template.md +++ b/docs/experiment-template.md @@ -10,6 +10,14 @@ day the question below was written, as `YYYY-MM-DD`, and it does not move afterwards, because the listing sorts by it. Add `Answer-Written` in the same change that writes the answer. +Add `Measurement-Commit` in that same change where the answer quotes a +measurement, with the object name of the commit the measurement was produced at, +written in full. It is neither here nor in `Answer-Written`'s position because a +template that ships a field filled in teaches every new record to declare a +value it does not have yet. The code the measurement ran against may be removed +later, and without this the answer keeps its numbers and loses the thing that +produced them while still reading as complete. + `Needs-Hardware` is what this experiment needs beyond the runner, in words somebody deciding whether to reproduce it can act on, or `none`. It starts at `none` here because that is the right answer for almost every experiment. A test @@ -19,8 +27,9 @@ under `internal/hardware`, in a file whose name ends directory says the other is refused. The format is `docs/decisions/0008-the-experiment-record.md`, as added to by -`docs/decisions/0015-an-experiment-declares-the-harness-it-needs.md`. This file -is a convenience and those records are the authority. +`docs/decisions/0015-an-experiment-declares-the-harness-it-needs.md` and by +`docs/decisions/0016-an-answer-names-the-commit-it-measured.md`. This file is a +convenience and those records are the authority. ## Question diff --git a/internal/check/check.go b/internal/check/check.go index fd8ee30..29063df 100644 --- a/internal/check/check.go +++ b/internal/check/check.go @@ -460,6 +460,7 @@ func walkExperiments(root string, res *Result) error { res.Refusals = append(res.Refusals, refuseQuestion(record, data)...) res.Refusals = append(res.Refusals, refuseState(record, data)...) res.Refusals = append(res.Refusals, refuseHeaderDates(record, data)...) + res.Refusals = append(res.Refusals, refuseMeasurementCommit(record, data)...) res.Refusals = append(res.Refusals, refuseDates(record, data, res.Now)...) res.Refusals = append(res.Refusals, refusePromotion(record, data)...) // The only rule here that reads the directory as well as the record, diff --git a/internal/check/measurement.go b/internal/check/measurement.go new file mode 100644 index 0000000..5dba8ce --- /dev/null +++ b/internal/check/measurement.go @@ -0,0 +1,98 @@ +package check + +import "fmt" + +// FieldMeasurementCommit is the object name of the commit a measurement in the +// answer was produced at. Record 0016 adds it and record 0013 makes it +// optional, as it makes every field added after it. +// +// It is declared here rather than beside the four record 0008 fixes, and that +// is the shape headerDateFields already argues for at its own list: a field a +// later record adds arrives with the check that reads it, so a checker built +// before the field is unaware of it and a field with no check has nowhere to +// hide. +const FieldMeasurementCommit = "Measurement-Commit" + +// RecordMeasurementCommitIsNotACommit refuses a measurement commit that is not +// the shape of an object name. +// +// An answer quotes a command and its output and says nothing about which +// version of the code the command ran against. Record 0004 lets that code be +// removed afterwards, and when it is, the answer keeps its numbers and loses +// the thing that produced them without reading as any less complete. +// +// The abbreviation is the value worth the fixture, because it is the mistake +// somebody actually makes: the short name is what git prints in a log, it is +// what a hand copies, it resolves on the machine it was copied on, and it stops +// resolving on a repository that has grown into a collision. It also cannot be +// told from a typo, since a short name that resolves to nothing and a mistyped +// one are the same string. +// +// A declared field with nothing after the colon is refused here too, for the +// reason the date refusal gives: record 0013 makes absence legal and an empty +// declaration a different statement, and a commit field with no value claims +// there is a commit rather than claiming there is none. +const RecordMeasurementCommitIsNotACommit = "record-measurement-commit-is-not-a-commit" + +// The object name lengths this accepts, and the reason there are two. +// +// Every object in this repository is named in the first of them today. The +// second is what the same repository would name its objects in under the other +// hash git can be told to use, and refusing it would make this check a decision +// about the object format, which record 0016 does not take and which is not a +// question about a measurement. +const ( + shortObjectName = 40 + longObjectName = 64 +) + +// refuseMeasurementCommit holds a measurement commit to the shape of an object +// name. +// +// WHAT IT CANNOT SAY, and this is the whole of the residual. Whether the object +// is in this repository, whether it is a commit rather than a blob, and whether +// the command quoted in the answer was ever run against it. The runner reads a +// checkout and opens no connection, and asking git any of those would cost it +// the dependency surface record 0001 chose. A value of forty hexadecimal +// characters that resolve to nothing passes here, so a green run says the name +// is resolvable in principle rather than that it resolves. +// +// A record whose bytes do not parse as a record is not judged here, for the +// reason every other header rule gives: nothing can read a field out of a file +// that has no header. +func refuseMeasurementCommit(path string, data []byte) []Refusal { + record, err := ParseRecord(data) + if err != nil { + return nil + } + + named, present := record.Field(FieldMeasurementCommit) + if !present { + // Absence is never a refusal. Record 0013 fixes that, and most + // records carry no measurement at all. + return nil + } + if isObjectName(named) { + return nil + } + + return []Refusal{{ + Property: RecordMeasurementCommitIsNotACommit, + Subject: path, + Detail: fmt.Sprintf("its %s is %q, and record 0016 writes an object name in full, which is %d or %d characters of lowercase hexadecimal", + FieldMeasurementCommit, named, shortObjectName, longObjectName), + }} +} + +// isObjectName says whether a value is the shape git names an object in. +func isObjectName(value string) bool { + if len(value) != shortObjectName && len(value) != longObjectName { + return false + } + for _, r := range value { + if (r < '0' || r > '9') && (r < 'a' || r > 'f') { + return false + } + } + return true +} diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/expected b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected new file mode 100644 index 0000000..466fe70 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected @@ -0,0 +1,4 @@ +directories 1 +records 1 +experiments present +decisions absent diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals new file mode 100644 index 0000000..7da7ef0 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/expected-refusals @@ -0,0 +1 @@ +record-measurement-commit-is-not-a-commit diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour b/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour new file mode 100644 index 0000000..69c87c6 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/near-neighbour @@ -0,0 +1 @@ +record-with-a-measurement-commit diff --git a/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md b/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md new file mode 100644 index 0000000..b19e620 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit-abbreviated/tree/experiments/one/EXPERIMENT.md @@ -0,0 +1,18 @@ +Slug: one +State: answered +Question-Written: 2026-08-01 +Answer-Written: 2026-08-02 +Measurement-Commit: e5067b1 + +## Question + +Does reading a tree of a thousand records cost more than a second? + +## Method + +Built the tree and timed the walk. + +## Answer + +No. The walk took eleven seconds, and the cost is in reading the records +rather than in walking the tree. diff --git a/testdata/cases/record-with-a-measurement-commit/expected b/testdata/cases/record-with-a-measurement-commit/expected new file mode 100644 index 0000000..466fe70 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit/expected @@ -0,0 +1,4 @@ +directories 1 +records 1 +experiments present +decisions absent diff --git a/testdata/cases/record-with-a-measurement-commit/expected-refusals b/testdata/cases/record-with-a-measurement-commit/expected-refusals new file mode 100644 index 0000000..e69de29 diff --git a/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md b/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md new file mode 100644 index 0000000..2d68aa5 --- /dev/null +++ b/testdata/cases/record-with-a-measurement-commit/tree/experiments/one/EXPERIMENT.md @@ -0,0 +1,18 @@ +Slug: one +State: answered +Question-Written: 2026-08-01 +Answer-Written: 2026-08-02 +Measurement-Commit: e5067b1d0f4a2c8b6e3a9d7c1b5f8a2e4d6c0b93 + +## Question + +Does reading a tree of a thousand records cost more than a second? + +## Method + +Built the tree and timed the walk. + +## Answer + +No. The walk took eleven seconds, and the cost is in reading the records +rather than in walking the tree.