Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/research/agent-procedure.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,24 @@ that a task is open. A deterministic seed is a reproducibility aid, not novelty.
Each card supplies a first-step command, expected evidence, a next experiment,
its connection to the bottleneck, allowed variations and scope-change triggers.

Before taking a source's next question, query its issue/PR identifier directly
in the dedicated PR body intake command. Author and creation-date filters can
isolate an account's recent contributions without GitHub's indexed body search:

```sh
python3 -B scripts/find_source_overlap.py --reference 6537 --pr-author petalas \
--created-since 2026-10-27 --include-closed --cache /tmp/source-overlap.json
```

Use the actual source number, login and UTC cutoff. `--reference` is repeatable
and excludes decimal/hash substring hits; ordinary positional terms retain
literal substring semantics. Author/date filters search only PR titles and
bodies. `--include-closed` adds a separately paginated closed/merged corpus.
The existing `find_overlap.py` CLI is unchanged. Online intake queries refresh
PR caches that lack author/creation metadata; offline queries with missing metadata fail
explicitly rather than reporting no matches. Filtered results are leads, not
evidence of semantic novelty, and do not replace live claim checks.

**Resolve the generic orientation command before running a historical card.**
If its `first_step.command` is `python3 -B scripts/research_baseline.py`, use the
current orientation command in [current priorities](current-priorities.md).
Expand Down
113 changes: 113 additions & 0 deletions research/source-reference-intake/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Source-reference intake for research overlap

A dedicated `scripts/find_source_overlap.py` CLI accepts reference queries
and filters PR bodies by author and creation date. This makes an account's recent work
on a proposed source visible directly in the fetched body corpus. Missing
filter metadata produces an explicit error, so an old cache cannot silently
hide a possible lead.

Claim type: workflow tooling with behavioral regression evidence. This
contribution was produced by Nick Petalas. No mathematical bound, kappa
improvement or predecessor-result verification is claimed.

## Use

From the repository root:

```sh
~/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/find_source_overlap.py \
--reference 6537 --pr-author petalas --created-since 2026-10-27 \
--include-closed --cache /tmp/source-overlap.json
```

Use the actual source number, login and UTC cutoff. `--reference` is
repeatable and applies identifier boundaries to numeric terms. Hashes and
decimal substrings cannot displace a real reference. Positional terms retain
their literal search semantics. The date filter uses creation, not update,
and includes the cutoff day's midnight UTC.

The dedicated intake CLI searches PR titles and bodies only. Closed/merged
bodies require `--include-closed`, with separate coverage and timestamps.
Local notes and Discussions are excluded. The existing `scripts/find_overlap.py`
remains byte-identical to the PR base; its functions are imported without
mutation. A filtered online query refreshes legacy PR metadata. An offline
cache missing the requested fields fails with an explanation.
`--pr-author` takes a login, rather than `@me` or a search expression.

## Evidence and reproduction

Eleven new regression groups cover identifier noise and spelling, literal
queries, inclusive dates and author case, optional closed bodies, legacy
offline errors, invalid metadata, creation/update separation, selective
online refresh and invalid CLI input. The existing 37 overlap groups also
pass. The registered replay runs all 48 groups in normal and optimized
Python, without network access.

```sh
~/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_package.py --package research/source-reference-intake
~/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/source-reference-intake/verify.py
~/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/check_research_commit.py --package research/source-reference-intake --check-source-history
```

Expected: `Ran 48 tests`, `OK`, `PASS_SOURCE_REFERENCE_INTAKE` and the
registered package PASS. [VALIDATION.json](VALIDATION.json) is generated
evidence bound to the tested script/test hashes. Default replay writes
nothing persistent. `verify.py --write NEW_PATH` records a fresh execution
and refuses to overwrite an existing path.

The dedicated CLI's offline query over the retained 587-open-PR snapshot,
fetched at 2026-11-03T14:10:56.715135+00:00, returned PR6571 as a source-6537 lead.
This only establishes the tool's retrieval behavior on that snapshot.
Neither its mathematics nor the source's result was evaluated. The eleven
synthetic regression groups make the behavior reproducible offline.

## Limits

Reference text is an overlap lead, not evidence of semantic novelty or
duplication. Work can share a model without citing its source. Current
claims, package questions and merged work still need inspection. Queries
do not provide an atomic server snapshot or a seven-day policy decision.
No selected frontier or frozen research package changes.

## Open next questions

Could a separately scoped provenance index also surface uncited package
reuse? First compare exact package-path references in PR bodies with a
complete changed-path inventory. Missing citations, renamed packages and
semantic equivalence remain blockers; they must not become automatic
novelty claims.

## Implementation and credit

The new filters traverse finite cached PR lists and validate dates before
comparing them. Unknown author/date values fail explicitly. The dedicated
CLI reuses the existing normalizer, identifier matcher, atomic cache writer
and PR deduplication functions. Its own PR-only cache refresh retains author
and creation metadata and preserves older evidence on failed publication.
The existing CLI and every other research package remain untouched.

Power of Ten rule 3 deviation: the Python cache/search command
allocates lists and dictionaries at runtime. The new pass adds linear
storage in the existing validated PR corpus size. A fixed-capacity service
port would require a separately declared corpus-size contract. Rule 9
deviation: unittest dispatch and mocks use their fixed test callbacks.
The replay runs one fixed subprocess with a 90-second timeout. Warning-strict
normal/optimized execution, AST compilation and the full affected suite
provide diagnostics; Ruff is unavailable in the prescribed environment.

Douglas Colkitt's overlap helper and its credited repository contributors
retain their attribution. Existing identifier matching and cache/pagination
machinery are reused. The new dedicated query and metadata filters, regression
fixtures and integration notes were produced by Nick Petalas. Apache-2.0.
Independent human review is unclaimed.

## CI compatibility repair

At head `e55541991331d578d88660dcbb34af60506a534d`, the initial hosted run
passed this package and four other jobs, but the historical overlap-path-index
replay failed with `source blob provenance mismatch: scripts/find_overlap.py`.
The shared-file edit in this PR caused that job to be selected. The correction
restores the exact PR-base helper bytes and moves the feature to the dedicated
CLI. The historical package and its source pins are unchanged. A regression
checks that the new CLI does not mutate imported legacy functions or record
shapes. New hosted confirmation remains pending.
18 changes: 18 additions & 0 deletions research/source-reference-intake/SOURCES.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"schema_version": 1,
"base_commit": "7c49fe32c55de02c2c873655c804b16e4c3fef0a",
"files": [
{
"path": "certificates/selected-result.json",
"usage": "provenance",
"git_blob": "d59b5035e7b4cbaa9d688a1d32dd0d6eb79c466a",
"sha256": "9b8d0eea95b3bf0dfcb7938a732a6439b87a2685dec2a5ab9e525b3b34debbfa"
},
{
"path": "docs/research/agent-task-catalog.json",
"usage": "provenance",
"git_blob": "3404fbdd5e7529a0674eef26114b5071e4586cec",
"sha256": "ead6cdc9d0af6d58e8a49744d8ec8c7c15e13d2ba93c276c434bcf9b51b93a81"
}
]
}
90 changes: 90 additions & 0 deletions research/source-reference-intake/TASK.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"schema_version": 1,
"task_id": "replay-runtime",
"priority_revision": 8,
"base_commit": "7c49fe32c55de02c2c873655c804b16e4c3fef0a",
"original_card": {
"id": "replay-runtime",
"question": "Can one measured replay bottleneck be sped up without dropping coverage?",
"starting_points": [
"scripts/run_isolated_tests.py",
"scripts/verify_replay_coverage.py",
"certificates/selected-result.json"
],
"methods": [
"Profile one isolated replay and optimize the measured bottleneck.",
"Design source-bound reuse with explicit dependency invalidation and negative controls."
],
"acceptance_gate": "Report before/after wall time on the same fixture, retained checks and invalidation behavior.",
"first_step": {
"command": [
"python3",
"-B",
"scripts/research_selected_baseline.py"
],
"expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
"inspect": "Resolve the current selected review, separate complex/ordinary BIT/admitted BIT/outer budgets, and compare the card against live handoffs before starting. Historical ceilings require exact source/model equivalence.",
"scope": "Source-binding orientation only; not a scientific replay or all-size proof. Materialize selected inputs if this checkout is sparse."
},
"first_experiment": "Measure one unchanged fixture, preserve check coverage, and test one optimization with invalidation controls.",
"bottleneck_link": "Faster trustworthy replays increase review throughput and let mechanism candidates be checked sooner.",
"allowed_variations": [
"Profile one stage or design one source-bound cache boundary."
],
"scope_change_triggers": [
"Skipping coverage or comparing different fixtures to claim speedup."
],
"overlap_terms": [
"replay runtime",
"cache invalidation",
"isolated tests"
]
},
"scope_status": "changed",
"actual_scope": "Add a dedicated PR-source intake CLI with reference, author and creation-date queries while leaving scripts/find_overlap.py byte-identical to the PR base.",
"scope_change_reason": "A source-reuse collision was already represented in the complete body cache but was missed by a targeted indexed search and a broad task query. This is an original coordination-tooling improvement, not another predecessor verification or a scientific replay speedup.",
"bottleneck_link": "Coordination side result: expose recent source reuse before spending work on a target. It changes no multiplication bound or supplier obligation.",
"budget_minutes": 60,
"stop_rule": "Stop after one working source-specific query with passing regression coverage and explicit cache-coverage limits. Do not extend to semantic novelty detection.",
"outcome": "completed",
"evidence": "Eleven new behavioral regression groups and the 37 existing overlap-named groups pass. The standalone CLI retrieves PR6571 from the retained body corpus. VALIDATION.json records generated hash-bound local execution.",
"author": "Nick Petalas",
"ai_assistance": "Independent human and formal proof-kernel review are unclaimed.",
"open_questions": [
{
"question": "Can a separately scoped package-provenance index surface uncited reuse without treating missing references as novelty?",
"why_it_matters": "Reference matching cannot detect a shared model whose source is not cited.",
"first_step": "Compare exact package-path references in PR bodies with a complete changed-file inventory and report coverage explicitly.",
"blocked_by": "Missing citations, renames and semantic equivalence require separate evidence; no automatic novelty claim is justified."
}
],
"claim_type": "workflow tooling with behavioral regression evidence",
"claim_url": "https://github.com/CrocSwap/integer-mult-bounds/discussions/1166#discussioncomment-19375429",
"scope_update_url": "https://github.com/CrocSwap/integer-mult-bounds/discussions/1166#discussioncomment-19375643",
"originality": "The CLI integration, author/creation metadata handling and regression fixtures are new. Existing numeric-reference matching, cache paging and source-history infrastructure retain their predecessor credit.",
"unrun_obligations": [
"Independent human review",
"Semantic novelty or duplication decisions",
"All mathematical predecessor replays and selected cold replay"
],
"orientation": {
"command": "/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_selected_baseline.py",
"output": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
"canonical_main": "7c49fe32c55de02c2c873655c804b16e4c3fef0a",
"priority_revision": 8,
"selected_pr": 6030
},
"overlap": {
"coverage": "Live handoffs, active Discussions, open/account PR lists, complete cached open-PR/Discussion pagination, main-tree distinctive and general-form searches, and merged author/filter searches.",
"limits": "Arbitrary diffs, uncited semantic counterparts and unpublished work are not fully covered."
},
"ci_repair": {
"failed_head": "e55541991331d578d88660dcbb34af60506a534d",
"run": "https://github.com/CrocSwap/integer-mult-bounds/actions/runs/41652301463",
"job": "replay (research/overlap-path-index)",
"error": "source blob provenance mismatch: scripts/find_overlap.py",
"cause": "The shared-file edit in this PR invalidated the historical consumer.",
"correction": "Restore the shared helper to the exact PR-base bytes and move the feature into scripts/find_source_overlap.py. No other research package or source pin is edited.",
"hosted_status": "pending"
}
}
35 changes: 35 additions & 0 deletions research/source-reference-intake/VALIDATION.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"claim_type": "workflow tooling with behavioral regression evidence",
"command": [
"/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12",
"-I",
"-W",
"error",
"-B",
"-m",
"unittest",
"discover",
"-s",
"tests",
"-p",
"test_research_overlap_*.py",
"-q"
],
"created_utc": "2026-11-03T14:39:20.279188+00:00",
"exit_code": 0,
"output": "----------------------------------------------------------------------\nRan 48 tests in 1.817s\n\nOK\n",
"platform": "macOS-27.0.1-arm64-arm-64bit",
"python": "3.12.15 (main, Oct 14 2026, 00:55:37) [Clang 23.1.3 ]",
"seconds": 1.900019499938935,
"sha256": {
"scripts/find_overlap.py": "c48d121d1fe4996a4c232b0d713d9163a1e47777ea3efe01e39bc36cec9aa053",
"scripts/find_source_overlap.py": "81788b200f4e2a195420016c289cc8387d3d7b531afdb60c2cb2f8aff3d489ef",
"scripts/research_package.py": "e1256016e7f51827d04fa2179255989f639ae1d97afaa9c16265f85a0c6abd70",
"tests/test_research_overlap_cache.py": "e49cf1c1707a16c0181f642ab8d969eb8e8689262277b42dba343d0a13295bad",
"tests/test_research_overlap_closed.py": "2a5d28e13fe57191702d86630763fdfd33842696077837d225c5677aea819b03",
"tests/test_research_overlap_intake.py": "152138d48ef155d33c4cb375a34200a98200b1ac5e1f073cf848096e1b631f7a",
"tests/test_research_overlap_local.py": "e7deb16c3f8d14523d31b568a7ea58f683cc56489af2009b2f5cada58a94c2da",
"tests/test_research_overlap_pagination.py": "b83e490c45f76b70eb394dd9706ff0b2f627211efe975909f9e1b40d8995f99c",
"tests/test_research_overlap_terms.py": "7dd29a1b44ab0c80b8b58a6a1eb7cff810dda942019754029b3bc38c9ff3e601"
}
}
44 changes: 44 additions & 0 deletions research/source-reference-intake/VERIFY.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
{
"schema_version": 1,
"status": "ready",
"python": "3.12",
"requirements": [],
"inputs": [
"scripts/find_source_overlap.py",
"scripts/find_overlap.py",
"scripts/research_package.py",
"tests/test_research_overlap_cache.py",
"tests/test_research_overlap_closed.py",
"tests/test_research_overlap_intake.py",
"tests/test_research_overlap_local.py",
"tests/test_research_overlap_pagination.py",
"tests/test_research_overlap_terms.py"
],
"commands": [
{
"argv": [
"{python}",
"-I",
"-W",
"error",
"-B",
"{package}/verify.py"
],
"timeout_seconds": 120,
"checks": "The focused suite: eleven dedicated intake groups and 37 legacy overlap groups, including legacy-module isolation and failed-refresh evidence preservation."
},
{
"argv": [
"{python}",
"-I",
"-O",
"-W",
"error",
"-B",
"{package}/verify.py"
],
"timeout_seconds": 120,
"checks": "The same complete suite with optimized Python, propagating optimization into the test subprocess."
}
]
}
48 changes: 48 additions & 0 deletions research/source-reference-intake/verify.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
"""Replay source-intake behavior and the focused overlap regression suite.

Produced by Nick Petalas. Apache-2.0.
"""

import argparse
from datetime import datetime, timezone
from hashlib import sha256
import json
from pathlib import Path
import platform
import subprocess
import sys
import time


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--write", type=Path, help="Write fresh evidence; refuse overwrite")
args = parser.parse_args()
root = Path(__file__).resolve().parents[2]
command = [sys.executable, "-I", "-W", "error", "-B"]
if sys.flags.optimize:
command.append("-O")
command.extend(["-m", "unittest", "discover", "-s", "tests",
"-p", "test_research_overlap_*.py", "-q"])
started = time.perf_counter()
result = subprocess.run(command, cwd=root, text=True, capture_output=True,
timeout=90, check=True)
elapsed = time.perf_counter() - started
print(result.stdout + result.stderr, end="")
if args.write is not None:
inputs = ["scripts/find_source_overlap.py", "scripts/find_overlap.py", "scripts/research_package.py",
*sorted(str(path.relative_to(root)) for path in
(root / "tests").glob("test_research_overlap_*.py"))]
evidence = {"claim_type": "workflow tooling with behavioral regression evidence",
"created_utc": datetime.now(timezone.utc).isoformat(),
"python": sys.version, "platform": platform.platform(),
"command": command, "seconds": elapsed, "exit_code": result.returncode,
"output": result.stdout + result.stderr,
"sha256": {name: sha256((root / name).read_bytes()).hexdigest() for name in inputs}}
with args.write.open("x", encoding="utf-8") as stream:
stream.write(json.dumps(evidence, indent=2, sort_keys=True) + "\n")
print("PASS_SOURCE_REFERENCE_INTAKE")


if __name__ == "__main__":
main()
Loading