diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c73f0b7..4478bc0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,14 +21,14 @@ This repository holds FunnelScript's research contracts, falsifiable hypotheses, 2. State a falsifiable hypothesis, a counterexample, and the invariant or acceptance criterion in the relevant file under `spec/`. 3. Record the test procedure and actual outcomes under `evals/`. Link a concrete application test or GitHub Actions run when the claim depends on implementation. 4. Update the accepted decision, debate, and hypothesis records if the design boundary changes. Keep the research and [application documentation](https://github.com/funnelscript/funnelscript/tree/main/docs) aligned. -5. Run the repository-local documentation contract test and verify the diff before submitting: +5. Run the same full research test suite as CI and verify the diff before submitting: ```bash -python3 -m unittest discover -s tests -v +python3 -m pytest -q git diff --check ``` -The documentation test checks relative links in canonical research documents. It does not verify external URLs, scientific claims, or the behavior of the application. Python's standard library is sufficient; no application dependencies are installed in this repository. +`pytest` is required for the full research regression suite, including the documentation-link contracts and other research oracles. `python3 -m unittest discover -s tests -v` is an optional **narrower** smoke check only. These tests do not verify external URLs, scientific claims, or the behavior of the application. Research tests require `pytest`; application dependencies are not installed in this repository. ## Compiler and runtime changes diff --git a/DEBATES.md b/DEBATES.md index b4ced75..9b60e99 100644 --- a/DEBATES.md +++ b/DEBATES.md @@ -1155,3 +1155,15 @@ Option A retained a caller-managed checkpoint but allowed accidental successful ## Phase 007G — Strict composability versus unconstrained scripting Selected: a bounded precedence-climbing expression tree, explicit named module exports, and file ingestion through the existing World event validation path. Rejected: arbitrary code evaluation, wildcard import merging, a package manager, a new event scheduler, implicit numeric widening or legacy/World engine unification. Falsifiers: incorrectly associated arithmetic, conflicting namespace bindings, invalid JSON acceptance, or altered seeded snapshot behavior. Research: spec/phase-007g-composable-language-v1.md. This is not an Accepted ADR. + + +## Recovery architecture alternatives (2026-11-02; unresolved) + +Accepted ADRs were locally mirrored through DEC-037 after a detected register drift, but this does **not** decide canonical ownership or provide cross-repository CI parity. Candidate A: research-owned accepted ADR register with generated app mirror. Candidate B: application-owned register with research carrying immutable references. Owner selection and a failing cross-repo drift test are required before treating R1 as complete. + +For Web Studio, either make a versioned client of authoritative World execution with independent trace fixtures, or retain only a clearly labeled seeded single-path illustration that rejects unsupported graphs. Local browser containment is not a research result, semantic engine parity, or a predictive market model. See application `docs/ARCHITECTURE_RECOVERY_AUDIT.md`; no new ADR was accepted. + + +## R3 resource envelope: counted queue admission versus streaming execution (2026-11-02) + +A finite processed-event cap does not bound the number of future events in the calendar or retained `RunResult` fields. Direct CPython 3.12/arm64 measurements reached 100,000 pending events and about 22 MB traced peak with `max_events=1`. Option A adds opt-in queue/history/intent admission quotas to the existing ordered heap, retaining v0 semantics whenever the limits do not bind. Option B introduces a separately typed streaming/aggregate result and checkpoint contract to reduce retained history for long jobs, but requires backpressure, restoration, and sink failure semantics. Neither option supplies a hard process-memory quota for arbitrary user payloads; neither is accepted yet. Proposed hypotheses H-RQ1/RQ2/RT1/RT2/RP1 and evidence: `spec/runtime-resource-envelope-v0.md`, `evals/runtime-resource-baseline-2026-11-02.md`. diff --git a/DECISIONS.md b/DECISIONS.md index 7691cda..6c884d9 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -38,11 +38,16 @@ | **DEC-028** | [Source-Scheduled Local Events Through the Existing Event Calendar](#dec-028-source-scheduled-local-events-through-the-existing-event-calendar) | 2026-10-31 | **Accepted** | Type-checked `emit` and bounded deterministic event scheduling | | **DEC-029** | [Type-Checked Pre-Event Guards for World Rules](#dec-029-type-checked-pre-event-guards-for-world-rules) | 2026-10-31 | **Accepted** | Deterministic conditional execution of typed World rules | | **DEC-030** | [Exact-Typed Local Event Records at Both Calendar Boundaries](#dec-030-exact-typed-local-event-records-at-both-calendar-boundaries) | 2026-10-31 | **Accepted** | Static message schemas, strict ingress and pre-enqueue checking | +| **DEC-031** | [Reproducible Source-Level Stochastic World Transitions](#dec-031-reproducible-source-level-stochastic-world-transitions) | 2026-11-01 | **Accepted** | Seeded stochastic World transitions | +| **DEC-032** | [Source-Imported World Modules and Initial Marketing Standard Library](#dec-032-source-imported-world-modules-and-initial-marketing-standard-library) | 2026-11-01 | **Accepted** | Explicit module boundaries and optional domain packages | +| **DEC-033** | [Tagged Git-Source Experimental Preview and Safe World Onboarding](#dec-033-tagged-git-source-experimental-preview-and-safe-world-onboarding) | 2026-11-01 | **Accepted** | Versioned preview distribution | +| **DEC-034** | [Repository-Specific Documentation Contracts for the Experimental Preview](#dec-034-repository-specific-documentation-contracts-for-the-experimental-preview) | 2026-11-01 | **Accepted** | Documentation link verification | +| **DEC-035** | [Curated, Named Agent Skills with an Explicit Vendor Boundary](#dec-035-curated-named-agent-skills-with-an-explicit-vendor-boundary) | 2026-10-31 | **Accepted** | Contributor-facing skill bundle integrity | +| **DEC-036** | [Explicit CLI Failure Taxonomy and Bounded JSON Event Inputs](#dec-036-explicit-cli-failure-taxonomy-and-bounded-json-event-inputs) | 2026-11-01 | **Accepted** | CLI error classification and bounded JSON input | +| **DEC-037** | [EOF-Safe Parsing and Exact-Typed Narrow World Arithmetic](#dec-037-eof-safe-parsing-and-exact-typed-narrow-world-arithmetic) | 2026-11-01 | **Accepted** | One binary operator, finite float outputs, controlled zero division | --- -| **DEC-037** | [EOF-Safe Parsing and Exact-Typed Narrow World Arithmetic](#dec-037-eof-safe-parsing-and-exact-typed-narrow-world-arithmetic) | 2026-11-01 | **Accepted** | One binary operator, finite float outputs, controlled zero division | - ## Detailed Architecture Decision Records ### DEC-001: Core-Outwards 5-Phase Implementation Strategy diff --git a/OUT_OF_THE_BOX.md b/OUT_OF_THE_BOX.md index 5adfd1b..7fe77f7 100644 --- a/OUT_OF_THE_BOX.md +++ b/OUT_OF_THE_BOX.md @@ -553,3 +553,13 @@ Potential future research would specify operator-controlled immutable anchors, s ## Phase 007G frontier — Future module package system and engine equivalence (unimplemented) Potential future work could compare semantic traces for deliberately paired legacy and World models or test explicit module package manifests. Neither a package resolver nor a single merged simulation runtime is implemented or approved by Phase 007G. + + +## Recovery frontier: trace bridges and bounded scale (2026-11-02; unapproved) + +A future versioned World/browser trace bridge and an independent heap/stream/aggregate resource benchmark could falsify semantic equivalence and memory scalability claims under adversarial event fan-out, changed agent ordering and withheld calibration data. Neither implementation nor a claim of 100k-agent readiness follows from the present local R1/R2 remediation. The owner must first choose authority, admission and migration contracts. + + +## R3 frontier: bounded calendar and aggregate retention (2026-11-02; not accepted) + +Count-based queue admission can reject event explosions but cannot cap arbitrary Python payload bytes or a blocking user reducer. Future research could compare typed aggregate-only checkpoints and child-process RSS limits, then falsify claims of replay, restoration and memory control under real multi-agent input. No implementation, implicit event dropping or public API promise follows from the current baseline. diff --git a/evals/runtime-resource-baseline-2026-11-02.md b/evals/runtime-resource-baseline-2026-11-02.md new file mode 100644 index 0000000..6e2f9bc --- /dev/null +++ b/evals/runtime-resource-baseline-2026-11-02.md @@ -0,0 +1,74 @@ +# Runtime Resource Baseline: Event Calendar vs. Processed Cap + +Status: **OBSERVED LOCAL DIAGNOSTIC, not a performance certification**. +Date: 2026-11-02. Companion proposal: +[spec/runtime-resource-envelope-v0.md](../spec/runtime-resource-envelope-v0.md). + +## Environment and procedure + +Application working branch `remediation/architecture-recovery-r1-r2`, commit +`77795dc`, CPython 3.12.13, arm64 Mac. The EventMachine source was unchanged +during these measurements. Commands ran through `uv run python -c` from the +application checkout, using stdlib `time.perf_counter` and `tracemalloc`. +Each reported case was sampled once, with garbage collection before initial +queue measurements; no warmup, repeated trials, process-level RSS measurement, +or hardware power control. Figures are observations, not expected CI timing. + +For the initial queue case: + +```python +import time +import tracemalloc +from funnelscript.core.event_machine import Event, EventMachine, Transition + +n = 100000 + +def noop(state, event, rng): + return Transition(state) + +tracemalloc.start() +t0 = time.perf_counter() +result = EventMachine(noop).run( + 0, + (Event(1.0, "later") for _ in range(n)), + seed=0, until=0, max_events=1, +) +elapsed = time.perf_counter() - t0 +_, peak = tracemalloc.get_traced_memory() +``` + +`Event` objects were allocated by the source generator inside the traced +window. `result.pending` was measured after successful completion. + +| Initial future events | Processed | Pending | Seconds | Peak traced bytes | +| ---: | ---: | ---: | ---: | ---: | +| 1,000 | 0 | 1,000 | 0.00366 | 221,044 | +| 10,000 | 0 | 10,000 | 0.03753 | 2,221,756 | +| 100,000 | 0 | 100,000 | 0.40469 | 22,126,740 | + +## Emission fan-out counterexample + +One due event at `time=0` reduced to `Transition(state, events=tuple( +Event(1.0, "future") for _ in range(100000)))`. The run used `until=0` and +`max_events=1`. Observed: **1 processed**, **100,000 pending**, +**0.46226 seconds**, **22,929,240 traced peak bytes**. The result was successful; +the processed cap did not constrain queue admission. + +For comparison, 100,000 independent immediately due events with a no-op +reducer and `max_events=100000` yielded **100,000 processed**, **0 pending**, +**0.85765 seconds**, **20,402,348 traced peak bytes**. + +## Interpretation and limitations + +These measurements confirm the implementation-level gap: `max_events` is a +processed-event cap, not a queue or total-memory bound. On this controlled +100k-event input, allocated pending history grew roughly proportionally with N. +No conclusion follows about arbitrary payload sizes, nested emitted objects, +worst-case callback behavior, resident set size, representative population +speed or handling millions of concurrent users. + +The next experiment must add independent queue admission and result retention +policies under a new reviewed contract, replay exact v0 ordering when limits do +not bind, and measure 1k/10k/100k events and agent identities with RSS and +adversarial fan-out. No production code, runtime semantics, public API or +accepted ADR was changed in this diagnostic. diff --git a/spec/runtime-resource-envelope-v0.md b/spec/runtime-resource-envelope-v0.md new file mode 100644 index 0000000..0891ee3 --- /dev/null +++ b/spec/runtime-resource-envelope-v0.md @@ -0,0 +1,135 @@ +# Runtime Resource Envelope v0: Queue Admission, Retention and Population Scale + +Status: **PROPOSED, NOT ACCEPTED OR IMPLEMENTED**, 2026-11-02. +Recovery stage: R3. Authority: accepted DEC-008/009/019/027/031 and +[event-kernel-v0](event-kernel-v0.md), +[population-identity-rng-v0](population-identity-rng-v0.md). +No existing public Python or World contract is changed by this document. + +## Problem and direct evidence + +`EventMachine.run(..., max_events=N)` limits processed due events only. It eagerly +loads every initial event into a heap, admits every valid emitted event and returns +fully retained `processed`, `intents`, and `pending` tuples. An `until` horizon +does not exempt future events from initial queue allocation. An event cap of one +therefore does not imply constant space. Observed on CPython 3.12.13/arm64: +100,000 future events, `until=0`, `max_events=1` produced 100,000 pending +events at a 22.1 MB traced peak. A single processed event emitting 100,000 +future events also returned 100,000 pending at a 22.9 MB traced peak. +See [measured baseline](../evals/runtime-resource-baseline-2026-11-02.md). + +`PopulationExecutor.run` executes one isolated `WorldExecutor.run` per enrolled +agent and retains every `RunResult`, identity partition and optional trace. +The currently supported adapter has no shared causal agent interactions or +incremental aggregate-only output contract. Existing vectorized *legacy* +`SimulationEngine` throughput does not benchmark this World path. + +## Contract axes and invariants + +For queue Q after each admission, processed history P, inert intents I and +declared limits `Lq`, `Lp`, `Li`: + +1. **Admission:** `|Q| <= Lq` throughout a successful bounded run. Apply the + same admission rule to initial events and transition-emitted events. A + count-exceeding input must raise a named resource-limit failure; silently + dropping events, ignoring a horizon-future event or truncating the return + tuple is forbidden. +2. **Processed budget:** retain accepted `max_events` semantics; a successful + run obeys `|P| <= max_events`. Queue limits and process limits are + independent: neither is evidence for the other. +3. **Result retention:** a full-result mode must return every processed event, + effect intent and pending event exactly once, in its documented order. + If configured history/intent limits are exceeded, fail explicitly; no + hidden sampling or summary substitution. Any aggregate-only mode has a + distinct versioned output type and cannot be mistaken for `RunResult`. +4. **Determinism:** if resource limits do not bind, `state`, `processed`, + `intents`, `pending`, event ordering and RNG draws must exactly match the + unlimited v0 run for identical source, inputs, seed and environment. +5. **Failure boundary:** a limit failure has a typed category, configured cap, + observed attempted count and the event boundary. No successful partial + `RunResult` is returned. A Python reducer may mutate external objects or + perform I/O; the kernel cannot roll back arbitrary reducer side effects. + Only kernel-owned queue/output publication can be atomic. +6. **Adversarial payload:** event-count caps do not cap Python object size, + cyclic graphs or arbitrary model state. A process-level RSS/memory guarantee + requires a separate policy/isolation mechanism. Any wall-clock deadline + checked only between reducers cannot interrupt a blocking Python callback. + +The kernel must remain domain-neutral: no wallet, budget, settlement, marketing +classification or inferred absorbing-state rule is introduced. + +## Falsifiable hypotheses + +- **H-RQ1:** For an iterable with `Lq+1` future events and `until=0`, + bounded admission rejects while consuming at most `Lq+1` entries. + Falsifier: it materializes the full unbounded iterable or returns a + success with missing future events. +- **H-RQ2:** One transition emitting more events than the remaining queue + capacity produces a resource failure, not a silently truncated calendar. + Falsifier: a pending event disappears or an effect is published as success. +- **H-RT1:** With nonbinding queue/history/intent limits, complete golden + tuples and fixed-seed stochastic traces are byte-for-byte equivalent to v0, + including same-time insertion and priority constraints. + Falsifier: adding limits changes the order/draws of an otherwise valid run. +- **H-RT2:** Full-result history and intent quotas reject oversized output; + a future aggregate-only API is distinguishable by its return type and + records exact counts without inventing missing traces. + Falsifier: a consumer receives incomplete `RunResult` indistinguishable + from a complete result. +- **H-RP1:** Population execution at N=1k,10k,100k identities has measured + elapsed time and RSS for full and proposed reduced-retention modes, with + identical final partition identities under nonbinding limits. No claim + follows before the 100k case completes on representative hardware. + Falsifier: missing/duplicated IDs, changed keyed RNG for matched agents, + or an unlabeled memory/throughput claim. + +## Implementation alternatives (decision pending) + +**A. Additive opt-in bounded run.** Reuse the existing heap, insertion ordinal +and `max_events`; add explicit per-run queue admission and separate retention +quotas with a typed failure contract. Default v0 behavior remains unchanged +for existing call sites. Advantages: smallest compatibility surface, easy +negative controls and rollback. Limitation: full pending/result retention still +scales with admitted event count; Python payload bytes are unbounded by counts. + +**B. Distinct streaming/aggregate runner.** Keep event ordering in one +authoritative scheduler while routing processed events through a declared +consumer and returning a versioned summary/checkpoint instead of full tuples. +Advantages: can avoid linear processed-history retention for long jobs. +Costs: replay/resume boundaries, output sink failures, backpressure, checkpoint +RNG state and multi-agent causal scheduling require their own accepted contracts. +Simply adding an iterator around existing full `RunResult` is not streaming. + +Recommended sequence: prove A against H-RQ1/H-RQ2/H-RT1/H-RT2 as a reversible +safety slice, benchmark both designs on explicit workloads, then choose if B is +necessary. Do not publish `max_pending` or retention parameter names, defaults, +error subclasses or summary schemas as stable APIs before acceptance. + +## Verification and workload matrix + +| Workload | Cardinality | Required evidence | +| --- | --- | --- | +| Future-only initial calendar | 1k, 10k, 100k events | `until=0`, `max_events=1`; elapsed, traced peak, RSS, pending | +| One-event fan-out | 1k, 10k, 100k emissions | Queue boundary failure and no lost accepted work | +| Fully due calendar | 1k, 10k, 100k events | Stable order, exact processed counts, deterministic replay | +| Mixed priorities / zero-delay loop | 1k and cap+1 | Original v0 golden order, explicit resource-limit precedence | +| Identity-isolated population | 1k, 10k, 100k agents | Seed mapping, final partitions, per-agent result retention, elapsed/RSS | +| Optional trace and effect intents | bounded/unbounded variants | Independent result budget, no silent truncation | +| Blocking/side-effecting reducer | controlled only | Document inability to enforce hard per-step deadline or external rollback | + +Use a fixed, documented source revision, CPython version, CPU, input generator, +event payload size and tracing settings. Run memory experiments in fresh +processes when measuring RSS, include warm/cold effects, and report exact failure +modes. Do not use fixed wall-time assertions in CI on shared runners. Stress +100k independent World agents only with an explicit resource cap/timeout and +operator approval; a synthetic local vectorized benchmark is not equivalent. + +## Stop, acceptance and rollback + +STOP if the design changes accepted event order, snapshot atomicity, default +World RNG, public `RunResult` completeness, arbitrary Python callback semantics +or a live-provider effect boundary without a separate reviewed ADR. No silent +default caps and no retention-dropping `RunResult` change. Final acceptance +requires reproducible RED/GREEN counterexamples, full conformance suites, +resource failure taxonomy, local and CI test evidence, owner-signed public +contract and a migration/rollback note. This proposal creates no release claim. diff --git a/tests/test_decision_index.py b/tests/test_decision_index.py new file mode 100644 index 0000000..5bf66a4 --- /dev/null +++ b/tests/test_decision_index.py @@ -0,0 +1,26 @@ +"""Decision-ledger integrity checks; does not assert cross-repository parity.""" + +import re +from pathlib import Path + + +DECISION_HEADER = re.compile(r"^### (DEC-\d{3}):", re.MULTILINE) +INDEX_ENTRY = re.compile(r"^\| \*\*(DEC-\d{3})\*\* \|", re.MULTILINE) + + +def test_every_detailed_decision_has_exactly_one_index_entry(): + ledger = Path(__file__).resolve().parents[1] / 'DECISIONS.md' + body = ledger.read_text(encoding="utf-8") + index = body.split("## Decision Index", 1)[1].split( + "## Detailed Architecture Decision Records", 1 + )[0] + detailed = body.split("## Detailed Architecture Decision Records", 1)[1] + + indexed_ids = INDEX_ENTRY.findall(index) + detailed_ids = DECISION_HEADER.findall(detailed) + assert len(indexed_ids) == len(set(indexed_ids)), "Duplicate decision index IDs" + assert len(detailed_ids) == len(set(detailed_ids)), "Duplicate detailed decision IDs" + assert set(indexed_ids) == set(detailed_ids), ( + f"Unindexed: {sorted(set(detailed_ids) - set(indexed_ids))}; " + f"missing details: {sorted(set(indexed_ids) - set(detailed_ids))}" + )