Skip to content

perf: add deterministic synthetic benchmark corpus - #379

Draft
seonghobae wants to merge 265 commits into
fix/public-markdown-resource-options-175from
feat/performance-envelope-375
Draft

perf: add deterministic synthetic benchmark corpus#379
seonghobae wants to merge 265 commits into
fix/public-markdown-resource-options-175from
feat/performance-envelope-375

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Scope and ownership

Refs #375. This branch is the established single-writer implementation lane for Inkspan's deterministic large-document performance evidence. Protected main remains the only shipped implementation authority. No SHA, merge-base, check result, review count, or workflow state recorded in this PR body is lifecycle authority; refetch those values at decision time.

The lane remains standalone, deterministic, provider-neutral, and network/model independent. It contains deterministic synthetic Markdown/Office fixtures, strict summary/comparison primitives, bounded Markdown and revision-evidence measurement harnesses, retained-memory settling analysis, demo-vendor chunking, an Office render duration/peak-RSS measurement primitive bound to the canonical synthetic fixture lock, and a one-command suite that composes Markdown + revision/transition-evidence/envelope-canonicalization + autosave enqueue/coalescing/commit measurements. Packed mode measures exact modules extracted from one verified npm artifact, rejects Markdown, HTML, or envelope inputs that do not match the committed profile lock, and records package/runtime/source/reference-hardware provenance.

This work does not by itself establish production performance budgets, a supported large-document envelope, complete packed-editor/browser/IME/Yjs/Office matrix acquisition, leak freedom, or release acceptance.

Transition scenario fidelity and prerequisite stack

This Draft branch inherits #402 through canonical serialization owner #176 and is stacked on fix/public-markdown-resource-options-175. Parent updates use ordinary merge history. The sole source conflict in that integration was resolved to match the owner exactly, preserving existing encoder reuse plus the owner's output bounds and hostile-option validation. Every benchmark, Office and demo-chunking delta remains intact. The prerequisite's CI, package and dependency changes remain owned by #402. None is shipped before protected integration.

The original transition-evidence-<profile> metric remains a no-op transition. --operation transition-changed --resulting-input <file> adds a distinct transition-changed-evidence-<profile> metric and enforces changed=true with unequal revision digests. The old scenario now requires changed=false with equal digests. Invalid/mismatched results, identical changed inputs, unsafe second-input paths, and output aliases fail before sample publication. The summarizer accepts the new ID while the comparator rejects cross-scenario comparisons.

The packed suite accepts the explicit resulting input with or without HTML serialization, validates its committed byte count/digest, and CI acquires both transition scenarios. Earlier fixture bytes/hashes are preserved. The added fixture appends one plain paragraph; envelope fixtures do not become rich editor list/table/image trees merely because their text contains Markdown. See benchmarks/README.md for the probe contract and limitations. This is measurement correctness and a separate synthetic baseline, not a speedup or a buyer-workload support claim.

Executable contract

Benchmark producers and fixture generators fail closed on unsafe symlink/non-directory ancestors, unsafe leaf targets, pre-existing hard-linked outputs, bounded-input violations, and unverifiable package/module provenance. Packed mode requires the exact package tarball digest and package identity, the active Node runtime identity, the source checkout identity, and a reference-hardware identifier.

benchmarks/run-current-suite.mjs additionally fails closed unless the benchmark checkout cleanliness guard succeeds before delegating to benchmarks/run-current-suite-core.mjs. This closes the false-provenance class where modified or untracked source could otherwise produce evidence labeled with an unchanged HEAD SHA. The failure is bounded and does not disclose dirty file paths.

office/benchmarks/measure_render.py accepts only a committed synthetic Office fixture whose exact byte count and SHA-256 match benchmarks/office-fixtures.lock.json. It requires a clean checkout and verified source revision, rejects unbounded iteration counts before inspecting caller-selected input, uses a fresh Python child process for each render sample, and records render duration plus process peak RSS with p50/p75/p95/max summaries and runtime/reference-hardware provenance. Ordinary evidence contains fixture identity/hash/size and measurements, never the document body or caller path. It performs no network, credential, service, database, or model operation.

Failure-contract / TDD lineage

A direct reproduction against the predecessor implementation established that git rev-parse HEAD alone cannot distinguish clean source from tracked or untracked worktree mutations. The narrow repair added a clean-checkout guard plus isolated temporary-repository contract tests for clean acceptance and dirty-state rejection; packed-suite tests exercise the delegated clean path. The test fixture is isolated from the repository checkout so parallel benchmark tests are not contaminated by a temporary dirty worktree.

The Office measurement contract was added test-first on the canonical performance branch: it requires lock-bound synthetic input, stable privacy-safe rejection of arbitrary/private content, bounded iteration work, isolated repeated samples, duration and peak-RSS evidence, and source/runtime/reference-hardware provenance. The immediately superseded test-only generation was cancelled before terminal hosted RED evidence, so it is lineage rather than passing evidence; current-head verification must be read live and predecessor/cancelled runs never transfer.

Earlier RED/GREEN iterations established output-symlink, hard-link, measurement privacy/resource, suite ordering, failure-atomicity, packed-artifact identity, runtime identity, source-SHA, summary/comparison, and retained-memory-analysis contracts. Historical workflow results document lineage only; they never transfer to a later head or base.

First-invocation measurement generation

The revision, Markdown/HTML and autosave producers record every requested operation invocation, starting with the first. They no longer execute a preliminary operation without recording its duration. Existing result validation, setup/timer boundaries and privacy-safe failure publication remain intact. This measures operations, not whole-process startup.

Version 2 introduced first-invocation accounting. New JavaScript latency samples now use contractVersion: 3, additionally deriving inputSha256 from the same bounded bytes captured for measurement; changed transitions retain a distinct ordered resultingInputSha256. No-file autosave identifies its prepared synthetic revision-evidence payload. Summaries preserve these identities in JSON and text. Legacy versions 1 and 2 retain their original shapes and meanings; the comparator rejects cross-version or mismatched-input pairs before computing a verdict, even when profile labels match. No historical evidence is backfilled. Office, corpus-lock and suite-inventory contracts retain their independent versions. See the measurement accounting contract. No editor API or supported-performance promise changes.

The first-invocation correction reproduced nine capped-call failures before the fix. A separate negative control removed only the generation-comparison guard and made both mixed-generation cases incorrectly succeed; restored tests reject both directions. These are measurement-correctness results, not latency improvement claims. Keep the historical raw evidence unchanged and start a fresh baseline for the new method.

Captured-input identity repair — 2026-09-06

The direct producer boundary could previously give different input documents comparable profile labels. This repair derives byte identities before the timer, preserves UTF-8 byte-order-mark identity, and verifies that later file replacement cannot relabel already captured input. Missing/malformed/extra/legacy-backfilled input identities and equal changed-transition identities fail closed. Hashes are not anonymization or authenticity proofs; no document content, path, new dependency, editor API change or timer change is introduced.

The source and failure lineage is recorded in captured-input identity research. Focused predecessor evidence is explicitly scoped: 86 checks passed at c4fba276, and 14 captured/prepared-input checks plus TypeScript passed at c7d829d9. The broader c7d829d9 performance run failed: 158 passed, 8 failed, 1 worker RPC error across 41 files. Its unchanged timeout and null-child-exit failures remain retained, not relabeled as success.

The current 75f195de full configured coverage run is terminal: 1,167 passed, 3 failed, 212 files, 674.14 seconds. The failures are unchanged deadlines in packed-artifact path stability and two corpus/source-provenance rejection tests; this is not green full coverage. The unchanged isolated run of the affected two files produced 6 passed and 2 different deadline failures; neither a timeout cause nor green full coverage is established. Current-head build and full package verification passed. Local Office verification passed 179 tests with configured statement/branch coverage and docstrings at 100% on one interpreter. The canonical source browser run passed all 70 tests across Chromium, Firefox and WebKit, including consensus, run e3377c8f-5f00-4f7c-932d-386269eefaad (package digest null). Packed-editor acceptance remains open; no predecessor result transfers. The Performance Evidence workflow is manually disabled and regular CI admission skips Draft PRs; neither was changed to obtain green checks. This remains Draft/Proposed and does not close #375 or establish the 20 ms target.

Historical source reconciliation and local acquisition — 2026-09-05

Later exact-head validation: the unchanged 75f195de full coverage run serialized without our other heavy validation jobs passed all 1,170 tests in 212 files, 154.25 seconds, with all configured source coverage metrics at 100%. Earlier failed full and isolated runs are retained. This single repeat neither proves their cause nor establishes a product speedup. Standalone benchmark subprocess code remains outside configured source coverage. New captured-input measurements and current packed-browser acceptance remain separate work.

On a51cd5f8220b1100d515cc8eb05685ce89a942a6:

  • 212 Vitest files / 1,131 tests passed; configured source coverage is 100% statements, branches, functions and lines.
  • Strict TypeScript, full build, all public package verification and performance workflow actionlint passed.
  • Project-local Office Python 3.14: 179 tests; source statements, branches and docstrings 100%.
  • Full source-harness browser suite: 70 passed across Chromium, Firefox and WebKit, including clipboard consensus; zero unexpected, skipped or flaky outcomes. This is separate from packed runtime acquisition.
  • Three sequential acquisitions × four locked profiles × nine operations × 25 recorded samples produced 108 complete sample files / 2,700 retained timed samples, with no failed process. A subsequent audit found one additional unrecorded preliminary operation in each producer process. Those historical samples retain that method; they must not be compared with the new first-invocation generation as an optimization result. No recorded sample was deleted.
  • The measured 0.7.0 archive SHA-256 is 7887c94822fc27ebd590627700c0c20c9b5c7f79d3b6c13b491fdb96d37b0321. Its Markdown/revision/autosave modules were byte-compared with the post-browser build. Source cleanliness and exact revision were checked before and after acquisition.
  • Runtime: Node 24.19.0; verified same Apple M5 10-core / 32 GiB hardware identity refhw-sha256-5e7a3cf2c807457c88b1f5e87e401548c7496b50c0917217dae566ab8cf9493e. This is a shared development host, not accepted release hardware; its earlier load snapshot was elevated. These measurements begin a new source-lineage diagnostic baseline, not a causal optimization comparison.

Changed-transition p95 values, milliseconds:

Locked profile Run 1 Run 2 Run 3 Median run p95
small 0.866 0.856 0.681 0.856
medium 6.651 3.230 2.944 3.230
large 34.267 12.677 15.055 15.055
stress 44.674 39.564 37.947 39.564

The stress scenario does not meet 20 ms. All slower runs remain in the denominator; neither a best run nor a changed/no-op scenario substitution is acceptance. The historical encoder-reuse 8.7% no-op result remains historical only. See the inherited canonical reconciliation record for retained-delta provenance and safety tests.

Configured TypeScript coverage excludes standalone benchmark subprocess code; its contract tests are not a claim of measured 100% subprocess coverage. Actual packed editor input/IME/mount/retained-memory acquisition, benchmark-script coverage, accepted support budgets and protected reference-hardware evidence remain open. Draft-admission skipped hosted jobs are non-passing; local results are not independent review or protected integration.

Remaining #375 acceptance work

This PR does not close #375 until protected evidence covers the applicable acceptance boundary. Remaining product work includes:

  • repeated acquisition from actual document workloads and the packed editor/runtime on accepted reference hardware; synthetic fixtures establish harness contracts, not buyer-workload performance;
  • trusted interaction, IME, mount/hydration, serialization, revision, autosave, Yjs, and print measurements at the source owner that can produce them without crossing Inkspan's host boundary;
  • full supported Office fixture/profile acquisition across DOCX/XLSX/PPTX runtimes, publication-path measurement where applicable, and evidence-derived conversion budgets; the current Office primitive is measurement infrastructure, not a published support envelope;
  • an accepted repeated-run retained-memory/leak/settling acquisition protocol rather than analysis of synthetic samples alone;
  • support budgets derived from measured evidence rather than raised after a regression;
  • accepted support-envelope architecture/operability/README claims;
  • protected activation and first acquisition from the implemented bounded PR smoke plus scheduled/manual full matrix; and
  • stable over-limit behavior wherever Inkspan owns the bounded API.

Browser/IME work, reference-host integration, release workflow, and organization-required workflow behavior remain with their established owners. Do not create competing source writers merely to make this performance lane appear complete.

Decision-time acceptance rule

Before any readiness, merge, release, closure, ownership, or support-envelope decision, independently refetch and reconcile at least:

  • protected main, this PR's exact head, its independently resolved live base, ancestry/divergence, mergeability, and changed paths;
  • all open Inkspan PRs/issues that can own or conflict with the affected source boundary;
  • formal reviews, last-push approval requirements, and unresolved review threads;
  • repository and organization-required workflow runs plus individual jobs and their actual checkout/source SHAs;
  • live rulesets/branch governance and required-review policy;
  • applicable package, browser, Office, coverage, security, SBOM, provenance, reproducibility, release, rollback, and operability evidence; and
  • releases/tags/registry state when making a publication or buyer-facing support claim.

Pending, queued, in-progress, skipped-required, cancelled, absent, neutral, failed, stale, predecessor, wrong-checkout, synthetic-source-only, status-only, model-only, or vacuous evidence is non-passing. Automated comments/reviews are technical input, not qualifying independent approval. Any material head/base/ruleset movement invalidates the corresponding decision evidence.

Do not self-approve, weaken gates, transfer predecessor evidence, fabricate release identity, create a competing CI/security writer, or represent branch behavior as protected-main shipped truth.

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Fresh packed-artifact acquisition completed for unchanged exact head 5701b7edfdbf76fc3537d81533d4e93944392b62.

Acquisition identity:

  • package: @contextualwisdomlab/cwl-editor@0.6.0
  • package SHA-256: 6d426e1538221c7830688fe6e50d1c6b7fb74184d1fa1f5a3928c448fb2f155f
  • runtime: node-24.19.0
  • reference hardware ID: refhw-sha256-8ff3ff73751b4e2a869d8d9e7792d94ee38562d1dbadbe9127a354a74218c54f
  • locked profile: small
  • samples per operation: 25

Observed p50 / p95 milliseconds:

  • Markdown serialization: 0.086916 / 0.337083
  • HTML serialization: 0.054042 / 0.235292
  • revision evidence: 0.166083 / 0.297458
  • transition evidence: 0.298291 / 0.566666
  • envelope canonicalization: 0.150959 / 0.293416
  • autosave enqueue: 0.073500 / 0.124209
  • autosave coalescing: 0.055292 / 0.127542
  • autosave commit: 0.001292 / 0.008500

The suite completed successfully from a clean detached checkout and exact packed artifact. This is a local reference snapshot, not an accepted support budget, cross-machine claim, protected-main activation, or release proof. Product hot-path changes remain with their existing source-owner PRs rather than being duplicated here.

Signed-off-by: Seongho Bae <me@seonghobae.me>

Commit-Message-Assisted-by: Claude (via Claude Code)
@seonghobae

Copy link
Copy Markdown
Contributor Author

Autoresearch experiment 5 kept at 9258bdbd: reuse the stateless native TextEncoder at the canonical envelope boundary. On the same Node 24.19.0 host, locked stress corpus, and 25-sample transition-evidence measurement, the parent 5701b7ed p95 values were 29.897/25.898/27.565 ms (median 27.565 ms). Candidate p95 values were 25.013/25.300/26.845/25.022 ms (median 25.161 ms), an 8.7% median reduction; the post-baseline candidate repeat remained 25.022 ms. Validation: 990 tests, 100% coverage, full build, and all packed-package consumer checks. This is local comparative evidence, not reference-hardware release evidence.

Preserve the no-op baseline and add an explicit before/after corpus, packed suite lane, and scenario correctness gates.

Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae
seonghobae changed the base branch from main to codex/fix-python-boundary-coverage September 5, 2026 07:27
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head local evidence: changed-transition probe and prerequisite integration

Source: 1879d20bccd25691e6da03752a0bb2693744282b. Non-force merge parent: #402 at 637b910d25dabb363e40d535c6d89f4a5beb8c6d. PR remains Draft and stacked behind that canonical owner; this is not protected-main or release evidence. Parent CI/package/lock/Python files have zero child delta.

  • TDD: the original measurement accepted both malformed no-op cases; the new changed-scenario/input-safety cases failed before implementation. Final contract tests validate explicit before/after inputs, changed/digest consistency, private-path redaction, symlink/hard-link/output preservation, unchanged fixture hashes, old CLI compatibility, and cross-scenario comparison rejection.
  • Final head: 198 Vitest files / 1,011 tests, with repository-enforced statement/branch/function/line coverage all 100%; typecheck, full build, and complete packed-package verification pass.
  • Python 3.14.6: 179 tests pass, 100% statement/branch coverage. An initial run overlapped the package verifier's temporary checkout directory and failed the clean-source guard in two tests; that non-acceptance run was discarded and the clean sequential run passed without weakening the guard.
  • Chromium 151.0.7922.34, Firefox 153.0, WebKit 26.5: source smoke 70/70, then actual extracted npm artifact 70/70. Packed consensus records this source SHA, package digest, Playwright 1.62.0, and browser lock digest 3f6b822f0b271b57cd255bc7efba677177b96079a304a43959260cb48e81d94b.
  • Package @contextualwisdomlab/cwl-editor@0.7.0 tarball SHA-256: cf2549759a7064bdc94e20aa4cab9b27b93debba36dea2733a001969b4678e4a. This is a local build, not a registry publication.
  • actionlint: no issues in the merged CI caller or performance workflow. No force push, gate change, approval, or main merge was performed.

Separate synthetic scenario baselines

One serial local acquisition on Node 24.19.0; 25 measured samples per operation/profile, one existing warm-up, all four full packed-suite profiles. Units: milliseconds. These are distinct workloads, not an A/B optimization comparison.

Profile No-op transition p95 Changed transition p95
small 0.558 0.610
medium 5.123 2.884
large 8.987 9.265
stress 33.720 34.320

The new scenario preserves the old metric's meaning and starts its own baseline. It does not establish the 20 ms target, realistic buyer-document latency, a rich-document-tree support envelope, physical-device IME performance, leak freedom, or hosted acceptance. Hosted current-head checks/reviews and protected prerequisite integration remain separate gates. Raw local receipts and the hardware fingerprint remain in the local evidence bundle; do not transfer predecessor results to a later head.

…euse

Stack performance PR #379 after canonical owner #176 at 94b5ca8. The sole conflicted source file now exactly matches the owner, retaining original encoder reuse, output bounds and hostile-option validation. All benchmark and chunking deltas remain intact.

Signed-off-by: Seongho Bae <me@seonghobae.me>
seonghobae added a commit that referenced this pull request Sep 5, 2026
Preserve PR #379 commit 9258bdb at canonical owner #176 without importing benchmark infrastructure. Retain current option validation and both byte-limit guards; add the RED/GREEN reuse and output-buffer isolation check.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae
seonghobae changed the base branch from codex/fix-python-boundary-coverage to fix/public-markdown-resource-options-175 September 5, 2026 12:40
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

First-invocation accounting and generation isolation — 2026-09-06

Candidate e74a8a73db8bbc6c9946b9dec04a59e866255179 preserves the canonical
#176/#402 stack and all existing performance-lane deltas. It adds no editor API,
dependency, timer budget, sampling reduction, retry allowance or runtime cache.

The earlier latency producers executed one preliminary operation that did not
enter the sample array. Nine capped-call cases reproduced that accounting
problem at 03345b029624f5c3945ebeec5d075db14d259b6e. The fix removes the
redundant invocations from revision, Markdown/HTML and autosave producers,
while retaining each scenario's validation and defined timer/setup boundaries.
Invalid first or later serialization results publish no sample file.

New JavaScript latency receipts use contractVersion: 2. The summarizer keeps
the generation in JSON and text; the comparator reads legacy version 1 but
rejects version 1/2 pairs before calculating a percentage. Test-first commit
b6d83403c530f8a7a0d0c45cfbe8954b1d9c849a failed 14 generation-support or
diagnostic assertions. A narrower negative control removed only the version
comparison guard: both mixed-generation comparisons incorrectly succeeded and
both tests failed. The guard is restored in the candidate.

Exact-candidate local verification:

  • 62 focused tests and 1,144 full tests in 212 files passed (--maxWorkers=2).
    Configured production source statement/branch/function/line coverage is 100%.
    Standalone benchmark subprocess scripts remain outside that instrumentation;
    these results are not a claim of their measured 100% coverage.
  • Full build and complete packed-package consumer verification passed. The
    retained tarball SHA-256 is
    e984397c0c307d14e8614748f4dcf60d1fd1e4542bdae0b398323800f0c8eeb9.
  • 70 source-browser cases passed with one worker, zero skipped, unexpected or
    flaky results, and zero global errors. The receipt binds this exact head and
    run 1d6ccacb-5c14-4ba6-9470-7d86bedf0934.
  • Office passed 179 tests and 100% statement/branch/docstring checks in the
    project-local macOS environment. This does not replace the Linux CI locked
    dependencies or supported hosted matrix.

The earlier 108 files / 2,700 timed samples remain unchanged. Their additional
unrecorded calls are now disclosed in the historical PR evidence. No timing
comparison across those generations is presented as a product improvement.
The harness contract,
TRD and gap baseline preserve the distinction between research infrastructure
and realistic buyer-workload acceptance.

Keep Draft and integrate the canonical parents first. These local results do
not establish current hosted checks, independent review, protected merge,
installed-tarball browser identity, accepted hardware, the 20 ms target, or a
supported large-document envelope. Refetch exact-head/base governance before
any lifecycle transition.

Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
… limits

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

seonghobae commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Latest output-consistency repair

Source-browser acquisition (not packed): first current 7fb20686 full
run retained45passed/5failed/20notrun in256.91501s. An empty release directory
left after prior archive preservation caused two receipt failures; it has now
been moved intact outside the checkout. Firefox independently timed out in
clipboard teardown and focus/print setup. Chromium/WebKit print frames were
directly inspected and show readable document content without toolbar chrome;
the Firefox frame was blank and provides no print acceptance. Failed evidence:
/private/tmp/inkspan-pr379-current-source.9MvKmC. The corrected full source run
then failed before executing tests: canonical webServer startup exceeded120s.
No source, guards, deadlines, workers or retries changed. That terminal result
is retained in /private/tmp/inkspan-pr379-source-clean.z4g2KA; no current
full-source acceptance or Firefox fix is claimed.

Current Office verification: same 7fb20686 head, Python3.14.6,179 tests
passed in17.36s. All598 configured statements and244branches covered with
zero misses; public docstring check100%. This is local single-interpreter
evidence, not the hosted version matrix. Logs are retained in the current
browser evidence directory below; the browser failure remains unresolved.

Current packed-browser acquisition: at 7fb20686, the full canonical
three-worker run with tracing retained 52 passed, 1 failed, 17 not run
(129.454936 seconds). Firefox hidden-office-eof-comment timed out in the
afterEach network-idle wait; incomplete corpus evidence was rejected and
consensus did not run. This is not current full-browser acceptance.
Actual final focus screenshots from all three engines were opened: readable
text and continuous, unclipped blue focus borders in the existing stylesheet
fixture. This does not establish full editor UX or WCAG conformance.
The package digest below was verified, JS/CSS matched extracted package bytes,
all failed receipts/traces were preserved, and the temporary archive copy was
moved outside the checkout to restore its clean source state. Evidence:
/private/tmp/inkspan-pr379-current-browser.0dd8gG.

Current package and new authored-input baseline: full build, independent
package consumer chain and fresh pack pass at 7fb20686. Archive SHA-256
f43b88b5e24eeb84da27ca58265de924f27c866f77266268350a69719cc8dcc1; measured
module SHA-256 8e76fe138c47ae6967f93b7ba38489021c85d1b65289be29ba66c19a6c824739.
All 12 fresh processes succeeded and retained all 300 samples on the same four
immutable authored documents from 75f195de (not current TRD or a representative
buyer corpus). Per-run nearest-rank p95 ranged 0.899084–4.839 ms; maximum
34.208125 ms. Ten samples exceeded 20 ms, all first invocations, none excluded.
The new output consistency check passed; four heading checks provide only
limited fidelity evidence. Both hashes rechecked after acquisition, worktree
clean. No speedup, page/k6 latency, full fidelity or release claim. Protocol,
raw samples and all summaries: /private/tmp/inkspan-pr379-stable-package.uxUKPU.

Current candidate is 7fb2068698bddd38264af169b4897f0eb4ee8802, normally
pushed and independently verified on the existing branch. The earlier
75f195de acquisition results below are predecessor evidence, not current
acceptance. The shared Markdown/HTML producer previously published samples
even when repeated identical input returned different strings.

RED 98f068a8f667447f947660e18283e8c5c7b236c1 reproduced four false passes
(both directions, changed second/third call after an empty first string).
Repair ea2e9982a067b2652dc413314854e4fdc9de30f2 retains the first measured
output and rejects inconsistent later output without publishing samples or
content. Comparison occurs after timing, adds no warmup and keeps first-call
accounting. Sixteen focused tests and TypeScript pass. Full current configured
coverage passed 1,174 tests across 212 files in 133.78 seconds, all configured
source metrics 100%. This does not claim 100% branch coverage of the external
benchmark scripts, which are exercised by subprocess contract checks.

The first full acquisition is retained: 1,159 passed, 15 failed in 228.52 seconds.
The clean-source guard rejected the sole untracked archive retained from earlier
packed-browser testing. Its independently verified bytes were moved intact to
the new evidence directory before the unchanged full rerun, not ignored or
deleted to weaken the guard. Current worktree is clean. No predecessor package
or browser receipt is promoted to current-head acceptance.

Harness README, TRD and the existing performance gap baseline distinguish
consistency from conversion fidelity. This additional validation retains an
output in memory and requires a fresh measurement baseline; no speedup, output
schema change, published editor behavior or predecessor-result transfer is
claimed. One sample cannot prove repeatability. New protocol and logs:
/private/tmp/inkspan-pr379-output-stability.nMI5LH.

Predecessor input identity repair evidence

Exact candidate: 75f195de0f8321848d544c3bb4cd33e0fefb9e8a.
Live parent: 94b5ca815749a7dd6bc071106eb685d2a193e4dd; Draft/open.

The configured full coverage run is terminal: 1,167 passed, 3 failed,
212 files, 674.14 seconds. It does not establish green full coverage.
The failures were unchanged deadlines in the packed-artifact path-stability
case and two corpus/source-provenance rejection cases. Raw log:
full-coverage-final.log.

An unchanged one-worker run of the affected two files then produced 6 passed,
2 failed
, 71.13 seconds. All three previously failing cases passed; different
runtime-ID and HTML-lock rejection cases exceeded their 5-second deadlines.
All three packed-suite identity variants passed, including ordered changed
inputs. This isolates no deterministic input-identity failure, but neither proves
the timeout cause nor replaces the full denominator. Raw log:
packed-isolated-75f195.log.

An additional unchanged full coverage run, serialized without our other heavy
validation jobs, passed all 1,170 tests in 212 files in 154.25 seconds, with all
configured source coverage metrics at 100%. The earlier failed runs remain
retained. One passing run does not prove the timeout cause or a product speedup;
standalone benchmark scripts remain outside the configured source coverage.

The current exact-head build and full pnpm verify:package are now terminal
and passed, including isolated ESM/CommonJS/strict TypeScript consumers.
Local Office verification passed 179 tests, with all 598 configured statements
and 244 branches covered and public docstrings at 100%. This is one local
interpreter, not the hosted supported-version matrix. The canonical source
browser run passed all 70 tests in 2.4 minutes across Chromium, Firefox and
WebKit, including consensus (run e3377c8f-5f00-4f7c-932d-386269eefaad,
package digest null). This is source evidence, not packed-editor acceptance.
Hosted Draft CI checks are skipped and no
current-head formal review was returned. The Performance Evidence workflow is
manually disabled; neither admission nor workflow state was changed.

All local logs remain under
/private/tmp/inkspan-input-identity-evidence.3O7Jac. No timeout, fixture size,
sample count, assertion, governance gate or historical sample was weakened.
This is measurement correctness work, not a demonstrated speedup, supported
large-document envelope, protected integration or release.

Actual authored-document baseline on the same head

The unchanged canonical README, PRD, TRD and CONTRACTS documents were measured
with the existing version-3 Markdown producer against a newly packed module.
All 12 fresh processes exited 0 and all 300 requested samples were retained,
including first invocations; there was no extra warmup. Per-run nearest-rank
p95 ranged from 0.569375 to 2.784667 ms. One sample, the first PRD invocation,
was 23.850375 ms; it remains in the evidence. These are operation-only timings,
not browser/page/k6 timings or a speedup claim. These four authored documents
are not a representative buyer corpus or the locked synthetic small fixture.

Archive SHA-256: 4151d7238d6e61acecc7b8ca4d39589c43b89eb2598b9c1edc5dd454ac3abc06.
Measured module SHA-256: 8e76fe138c47ae6967f93b7ba38489021c85d1b65289be29ba66c19a6c824739.
Input hashes, clean exact head, and both artifact hashes were verified after
acquisition. Separate title-preservation and repeated-output checks passed;
they do not establish complete output fidelity. Protocol, every raw sample,
individual summaries and limitations are retained under
/private/tmp/inkspan-real-docs-evidence.oHOrKn.

Full extracted-package browser validation subsequently passed all 70 tests
across Chromium, Firefox and WebKit, including consensus, on this same head.
Receipt run a4ff7ead-fcfe-45eb-8f17-b510534032ac records the exact archive
digest above in all engines. An initial 70-test pass had null package identity
because the existing receipt helper could not find the archive under release/;
that incomplete evidence is retained. Copying the same archive into the
previously absent expected directory and rerunning unchanged tests supplied
the linked evidence. No assertion, deadline, source or package bytes changed.

Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: performance Performance, resource use, scalability, or benchmarking enhancement New feature or request priority: medium Normal-priority or P2 work status: draft Draft pull request type: feature New or expanded product capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: define and enforce a large-document latency and memory support envelope

1 participant