Skip to content

docs(195): restructure roadmaps along the contract/implementation layers - #197

Merged
pdlourenco merged 2 commits into
mainfrom
docs/195-roadmap-restructure
Jul 29, 2026
Merged

pdlourenco merged 2 commits into
mainfrom
docs/195-roadmap-restructure

Conversation

@pdlourenco

@pdlourenco pdlourenco commented Jul 29, 2026 •

Copy link
Copy Markdown
Owner

Implements the maintainer-approved plan in #195. Docs-only: no spec/ edits, no code changes.

What changed

  • docs/roadmap.md rewritten (path unchanged) as the language-neutral catalogue: the sid + Domain + Method convention including the snake_case translation rule, one merged capability table (MATLAB | Python | Julia, status per port), reserved names, online/recursive COSMIC, and out-of-scope. The "Result Struct" section is deleted — spec/SPEC.md owns output fields.
  • Layer-2 material relocated: Octave compatibility rules → matlab/CONTRIBUTING.md; spec→NumPy translation notes + plotting conventions → python/CONTRIBUTING.md. Merged, not duplicated — the lambda_ and RNG notes already existed there and were left alone. The MATLAB↔Python private-helper mapping is dropped, not relocated (it was already drifting; helper decomposition is an implementation detail each port may choose).
  • Both phase logs archived under docs/plans/ behind historical-disclaimer headers that name the "MATLAB is ground truth" claim and the copy-porting workflow as superseded doctrine.
  • docs/roadmap_python.md and docs/TODO.md deleted. TODO's one live item (multi-language repo refactor, deferred v2) folded into out-of-scope.
  • ADR-0006 records the restructure; README.md, matlab/README.md, python/README.md retargeted and relabelled.

Acceptance criteria

# Criterion Evidence
1 roadmap language-neutral, no spec duplication catalogue is capability-keyed; only result. mentions are the naming example
2 roadmap_python.md / TODO.md gone, content relocated deleted; live content in roadmap + language guides + archives
3 Phase logs archived with disclaimers docs/plans/2026-07-29-*.md
4 No living document claims MATLAB is ground truth git grep — remaining hits are the prohibition + ADR quoting it
5 All inbound references resolve 71 relative links + 4 anchors checked, 0 broken
6 ADR in same PR ADR-0006 + index entry

Verification

  • Archive fidelity: the archived phase-log sections are byte-identical to the originals (verified by diff); the remaining content was relocated or deleted as itemised above. Each archive header now names exactly which sections it reproduces and where everything else went. History archived, not rewritten, per the constraint.
  • Catalogue checked against code — every listed MATLAB name exists in matlab/sid/*.m, every Python name in python/sid/*.py. (The old roadmap listed four private helpers as public utilities; that's corrected.)
  • Link/anchor checker run repo-wide.

Pre-push self-review — one blocker found and fixed

A fresh-context reviewer subagent (seeded with docs/REVIEW_CONTEXT.md) caught that my first draft's out-of-scope rewrite invented a "non-goal vs deferred" split, silently reclassifying five items that spec/SPEC.md §8.14 lists as "Deferred Extensions" into permanent non-goals — an unauthorized scope change in the very document governance treats as authoritative for scope, and self-contradictory on its face (it called alternative LTV algorithms a non-goal while noting the 'Algorithm' parameter is ready for them). Restored to the flat "out of scope for v1.0" framing with the spec named as normative.

Also fixed from that review: an ambiguous six-column table header with duplicate column names (now one column per port), an inaccurate "~95%" figure in the ADR, and stale roadmap labels in matlab/README.md and tests.yml.

Known not-closed (deliberate)

The copy-by-copy porting workflow survives in python/CONTRIBUTING.md §"Porting Workflow" — it instructs contributors to read and translate the MATLAB source, never mentioning spec/SPEC.md. This PR removes the ground-truth claim but not that practice: rewriting it is a doctrine change beyond a docs restructure and deserves its own review. Recorded honestly in ADR-0006 Consequences ("this ADR narrows the contradiction, it does not eliminate it") and tracked in #196.

Refs #195

🤖 Generated with Claude Code

The two roadmap files cut across sid's development layers instead of along
them, and neither was a roadmap any more: both were largely checked-off
execution logs. Worse, docs/roadmap_python.md stated "The MATLAB code in
matlab/sid/ is the ground truth for numerical behaviour" -- directly
contradicting ADR-0001 / CLAUDE.md §3 in a living document.

- docs/roadmap.md is rewritten (path unchanged) as the language-neutral
  catalogue: the sid + Domain + Method convention incl. the snake_case
  translation rule, ONE merged capability table (MATLAB | Python | Julia,
  status per port), reserved names, online/recursive COSMIC, out-of-scope.
  The "Result Struct" section is deleted -- spec/SPEC.md owns output fields.
- Layer-2 material moves to the language guides: Octave compatibility rules
  into matlab/CONTRIBUTING.md, the spec->NumPy translation notes and plotting
  conventions into python/CONTRIBUTING.md (merged, not duplicated -- lambda_
  and RNG notes already existed there and were left alone). The drifting
  MATLAB<->Python private-helper mapping is dropped, not relocated: helper
  decomposition is an implementation detail each port may choose.
- Both phase logs archived under docs/plans/ VERBATIM (verified byte-identical
  by diff) behind historical-disclaimer headers naming the ground-truth claim
  and the copy-porting workflow as superseded doctrine.
- docs/roadmap_python.md and docs/TODO.md deleted; TODO's one live item
  (multi-language repo refactor, deferred v2) folded into out-of-scope.
- ADR-0006 records the restructure; README.md/matlab/README.md/python/README.md
  references retargeted and relabelled.

Pre-push review (fresh-context subagent) flagged one blocker, fixed here: the
first draft's out-of-scope rewrite invented a "non-goal vs deferred" split that
silently reclassified five SPEC §8.14 *Deferred Extensions* as permanent
non-goals -- an unauthorized scope change. Restored to the flat "out of scope
for v1.0" framing with the spec named as normative. Also fixed: ambiguous
six-column table header (now one column per port), an inaccurate ~95% figure in
the ADR, and stale roadmap labels in matlab/README.md and tests.yml.

The copy-by-copy porting workflow survives in python/CONTRIBUTING.md and is NOT
closed by this PR -- rewriting it spec-first is a doctrine change beyond this
restructure. Recorded in ADR-0006 Consequences and tracked in #196.

Refs #195

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Review

Per docs/REVIEW_CONTEXT.md (verification + validation). Mechanical checks ran against the branch head (36884bc): archive-fidelity diffs, catalogue-vs-code cross-check, anchor/pointer resolution, and a repo-wide grep for the acceptance criteria. All 12 changed files were read in full. Disclosure: this reviewer session authored the plan in #195 (not this diff) — verification of the implementation is independent, but validation of the plan itself shares that session's blind spots.

Summary

Implements #195 faithfully and, in several places, more carefully than the plan asked: docs/roadmap.md is now genuinely language-neutral (naming convention incl. the snake_case rule, one capability-keyed catalogue, forward-looking material only), layer-2 content landed merged-not-duplicated in the language guides, both phase logs are archived behind disclaimers that name the superseded doctrine explicitly, and ADR-0006 is honest about costs and about what it does not close (#196). Direction is right; ready to merge as far as this review is concerned, with two below-the-line items for the author's judgment.

What was verified (all pass)

  • Archive fidelity: both archived bodies diff byte-identical against the corresponding sections of the originals; the only content not carried over is exactly the sections relocated into living documents (Octave rules, Technical Notes, out-of-scope lists — all accounted for at their destinations, item by item).
  • Catalogue correction is legitimate: the four "Utility Functions" the old roadmap listed as public (sidCov, sidHannWin, sidWindowedDFT, sidUncertainty) are indeed in matlab/sid/private/ — dropping them from the public catalogue fixes a pre-existing error.
  • Acceptance criterion 4 holds: remaining "ground truth" hits on the branch are the prohibitions (CLAUDE.md, CONTRIBUTING.md, REVIEW_CONTEXT.md, ADR-0001) and ADR-0006 quoting the removed claim.
  • Pointers resolve: spec/SPEC.md §8.14 "Deferred Extensions" and §8.12.4 (cited by the MATLAB archive header) exist; the #reserved-word-handling, #test-categories, #testing anchors exist; the root CONTRIBUTING.md does document scripts/local-ci as referenced from the new Octave section; the GitHub slug #onlinerecursive-cosmic is correct; docs: make python/CONTRIBUTING.md "Porting Workflow" spec-first #196 exists and matches ADR-0006's "not closed" consequence.
  • The pre-push self-review finding (the invented "non-goal vs deferred" split) was the right thing to catch and the fix — flat v1.0 list with §8.14 named as normative and "spec wins" — is the correct resolution.

Issues to address before merge

None blocking.

  1. Non-blocker — the Reserved-names and Planned tables now mint Python spellings that never existed anywhere (tf_arx, tf_armax, ss_n4sid, ts_ar, ts_arma, nyquist_plot, pole_plot, ltv_disc_init/_update/_smooth). They follow the translation rule mechanically and I have no objection to any of them — but reserving names is exactly what the catalogue makes binding (REVIEW_CONTEXT "catalogue drift"), and these are additions, not relocations. Flagging so their blessing is a conscious act of this merge, not a side effect.
  2. Non-blocker — two silent drops relative to the originals. (a) The old roadmap's "Package Structure" + "Usage" sections were dropped without relocation or a line in ADR-0006's costs (the "Result Struct" deletion and the helper-mapping drop are recorded; this one isn't). The content was stale and duplicates README/DESIGN territory, so dropping is right — a half-sentence in the ADR would make the record complete. (b) "Julia port" silently moved from the old out-of-scope list into the catalogue as a planned column — consistent with CLAUDE.md/README, and the old entry was itself stale, but same remark: it's a reclassification, not a move.

Follow-up suggestions

  • docs/analyses/2026-07-12-repo-wide-review.md:171 mentions docs/TODO.md in prose (not a link, so criterion 5 is technically met). Analyses are dated records — no change needed, just noting it was seen.
  • The ADR's "generate the catalogue from source" alternative is worth revisiting once docs: add MkDocs Material documentation site #110's docs tooling lands: scripts/build_matlab_api.py already enumerates the public MATLAB surface, so a cheap CI assertion (every public function appears in the catalogue) would mechanize the drift check the ADR currently leaves to review time.

Verdict

Approve (items 1–2 at the author's discretion; neither warrants a re-review). CI is green across all 17 checks. Merging remains the maintainer's call per repo policy.


Generated by Claude Code

@pdlourenco pdlourenco left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review — roadmap restructure (#195)

Reviewed with fresh context, verifying each acceptance criterion myself rather than reading the PR's evidence column. Verdict: Approve. All six criteria hold. One wording correction on a verification claim, and two notes.

Acceptance criteria — independently verified

# Criterion My check
1 roadmap language-neutral, no spec duplication "Result Struct" section absent; the only result. occurrence is the snake_case translation example (result.NoiseSpectrum -> result.noise_spectrum) — legitimately a naming rule, not an output-field table
2 roadmap_python.md / TODO.md gone, content relocated both deleted; repo-wide grep finds one surviving reference, and it is correctly left alone (see note 1)
3 Phase logs archived with disclaimers both carry "Historical document — do not treat as current guidance"; the Python one names the ground-truth claim and the copy-porting workflow as superseded, with the offending preamble reproduced under a blockquote rather than edited
4 No living document claims MATLAB is ground truth 7 grep hits, all legitimate: the prohibition (CLAUDE.md §3, REVIEW_CONTEXT.md principle 2, ADR-0001), the ADR/index describing the removal, and 2 inside the disclaimed archive
5 All inbound references resolve independent walk of 237 relative links across every .md: 0 broken in this PR's scope (10 apparent hits are my regex matching math notation like (k)/(ω) in spec/EXAMPLES.md — pre-existing, and a file this PR doesn't touch)
6 ADR in same PR ADR-0006 present; I also ran #128's parallel-track scan — no other in-flight PR claims 0006, so the number is safe

The pre-push blocker fix is genuinely fixed

This was the highest-consequence item, so I checked it against the normative source rather than the description. The out-of-scope section is now flat "Deferred, not rejected", names spec/SPEC.md §8.14 as normative, and adds an explicit tie-break ("if the two disagree, the spec wins"). All seven of §8.14's deferred items appear in the roadmap list, and grep -i 'non-goal\|never be\|will not be supported' returns nothing. The unauthorized reclassification is fully reverted — and the tie-break clause is a better outcome than the original text, since it makes the precedence explicit rather than implied.

A real accuracy win the PR undersells

The old roadmap advertised sidCov, sidHannWin, and sidWindowedDFT as public "utility functions" (old lines 58–60 and 163–165). I confirmed all of them live in matlab/sid/private/ — they were never public API. The new catalogue omits them. That's not just tidying: the previous document was inviting users to call functions MATLAB's private/ mechanism makes uncallable.

Relatedly, dropping the MATLAB↔Python private-helper mapping rather than relocating it is the right call for a reason worth stating: a port-to-port helper correspondence table is exactly the artifact ADR-0001 rejects. Ports conform to the spec, not to each other; helper decomposition is each port's business. Deleting it removes a standing invitation to drift.

Correction — the archive-fidelity claim overstates its scope

The PR states: "both archived bodies diff byte-identical to the originals." Read literally that's not so — docs/roadmap_python.md was 491 lines; the archive is 311.

What is true, and what I verified: the phase-log portion (Porting Workflow → Timeline, original lines 205–472) is byte-identical to the archive's lines 46–311 — the only difference across 266 lines is a trailing --- separator dropped at the extraction boundary. The other ~180 lines were relocated (naming/catalogue → roadmap.md, technical notes → python/CONTRIBUTING.md) or deliberately deleted (Result Types, private helpers), all as the PR describes elsewhere.

So the substance — history archived, not rewritten — holds completely. Only the phrasing overstates. Suggest: "the archived phase-log sections are verbatim; the remaining content was relocated or deleted as itemised above." Worth the edit because precision about verification claims is now this repo's house standard, and a future reader auditing that sentence against wc -l would think something was lost.

Notes (no action)

  1. The one surviving docs/TODO.md reference is in docs/analyses/2026-07-12-repo-wide-review.md — an immutable dated analysis. Per the #154 convention (supersede, don't edit) leaving it is correct. Flagging so a future link sweep doesn't "fix" a historical record.
  2. Deferring the copy-porting workflow to #196 is right. python/CONTRIBUTING.md §"Porting Workflow" still says read-MATLAB-and-translate, and rewriting it is a doctrine change, not a docs restructure. Recording it in ADR-0006's Consequences as "narrows the contradiction, does not eliminate it" is the honest framing — that sentence is what makes the deferral legitimate rather than an omission.

🤖 Generated with Claude Code

Review of #197 correctly caught that "extracted verbatim from
docs/roadmap_python.md" overstates its scope: that file was 491 lines, the
archive is 311. The phase-log sections ARE byte-identical (re-verified by diff
after this edit) -- but the header, read literally, implied the whole file was
reproduced, so a future reader auditing it against `wc -l` would suspect
content was lost.

Both archive headers now name the exact sections reproduced verbatim, and
itemise where everything else went: relocated (naming/catalogue -> roadmap.md,
Octave rules -> matlab/CONTRIBUTING.md, technical notes ->
python/CONTRIBUTING.md, out-of-scope -> merged) or deliberately deleted (Result
Struct / Result Types as spec duplication, stale package trees, and the
private-helper mapping ADR-0001 rejects).

No archived content changed -- headers only.

Refs #195

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pdlourenco

Copy link
Copy Markdown
Owner Author

Correction accepted and fixed in cad609c — and you were right that it mattered.

On the archive-fidelity claim. I verified your arithmetic: the former docs/roadmap_python.md was 491 lines, the archive is 311, and the extraction was original lines 205–470 → archive 46–311. So "both archived bodies are byte-identical to the originals" was, read literally, wrong. The --- at original line 471 is the section separator at the extraction boundary; not carrying it is correct (it divided the phase log from "Technical Notes"), but that's a detail my phrasing papered over rather than disclosed.

I fixed it in the durable place first, not just the PR body: both archive headers previously said "extracted verbatim from <file>", which is exactly the ambiguity you caught — it reads as the whole file. They now name the precise sections reproduced verbatim and itemise where everything else went (relocated: naming/catalogue → roadmap.md, Octave rules → matlab/CONTRIBUTING.md, technical notes → python/CONTRIBUTING.md, out-of-scope merged; deleted: Result Struct / Result Types as spec duplication, stale package trees, and the private-helper mapping). The Python header states the 491 → ~266 line arithmetic outright, so the audit you performed now resolves from the document itself instead of requiring git show.

Re-verified after the header edit: both phase-log bodies still diff clean, and 73 relative links + 4 anchors resolve. No archived content changed — headers only.

On your two notes — both correct, and I'd already reasoned the same way:

  1. The docs/TODO.md mention in docs/analyses/2026-07-12-repo-wide-review.md is prose in an immutable dated record, and it's a backticked filename, not a link — so nothing dangles. Left deliberately per the Adopt analyses/ doc-type convention (seed ADR-0010) #154 supersede-don't-edit convention. Your flag for future link sweeps is the right precaution.
  2. Agreed on docs: make python/CONTRIBUTING.md "Porting Workflow" spec-first #196. The ADR Consequences sentence is doing real work there — without it the deferral would read as an omission rather than a scoped decision.

On the utility-functions find: worth recording that this was a live correctness bug in the docs, not tidying — sidCov, sidHannWin, and sidWindowedDFT sit in matlab/sid/private/, so the old roadmap was advertising as public API three functions MATLAB's private/ mechanism makes uncallable from user code. Anyone who tried would have hit an undefined-function error.

Ready for merge when you are.

@pdlourenco
pdlourenco merged commit 71be61c into main Jul 29, 2026
12 checks passed
@pdlourenco
pdlourenco deleted the docs/195-roadmap-restructure branch July 29, 2026 21:41
pdlourenco added a commit that referenced this pull request Jul 29, 2026
Adapts the docs-site PR in place per the maintainer decision on #110, rather
than closing or rebuilding it. The PR's content was essentially intact -- almost
everything regenerates from source -- so this is a restructure plus the four
blockers, not a rewrite.

Merge resolution:
- .gitignore: union of both sides.
- docs/TODO.md and docs/roadmap_python.md: the instruction said to keep them at
  their original paths, but #197 (merged ~3h after the review was posted) had
  deleted TODO.md outright and archived roadmap_python.md to
  docs/plans/2026-07-29-python-port-phase-log.md. Accepted main's side for both:
  nothing to restore, nothing to publish. This satisfies the "no dev-tracking on
  the public site" blocker more completely than restoring them would have.
- docs/roadmap.md: restored to its canonical path (the PR had moved it to
  docs/about/roadmap.md), carrying main's rewritten language-neutral catalogue.
  CLAUDE.md, docs/REVIEW_CONTEXT.md and CONTRIBUTING.md reference it by path.

Restructure -- site source out of docs/:
- All site source moved docs/ -> docsite/ (index.md, about/, api/, concepts/,
  examples/, getting-started/, hooks/, javascripts/, spec/, stylesheets/).
  docs/ now holds only internal engineering docs: DESIGN.md, REVIEW_CONTEXT.md,
  roadmap.md, decisions/, analyses/, plans/. Future internal docs can neither
  break the site build nor leak into the site.
- mkdocs.yml: docs_dir, edit_uri, snippets base_path, hooks paths retargeted.
  Include paths needed no change -- docsite/ sits at the same depth docs/ did.
  Generator scripts needed no change -- they use REPO_ROOT plus virtual
  mkdocs_gen_files paths, so they are source-dir agnostic.
- Removed roadmap/python-roadmap/TODO from the public nav, with a comment
  recording that publishing any internal doc is a separate deliberate decision.

Blockers:
- docs.yml now builds with --strict, so link rot fails the build instead of
  shipping silently (this is what the 39 pre-existing link warnings were).
- docs.yml runs on pull_request, so a PR that breaks the docs is caught before
  merge; deploy is split into its own job gated on main and never runs for a PR.
  Deleted the hardcoded claude/plan-sid-docs-website-vFMRn trigger.
- permissions scoped per job (workflow default {}, contents:read to build,
  contents:write only to deploy) instead of workflow-wide contents:write.

Non-blockers taken:
- Changelog page now mirrors the root CHANGELOG.md (added in 0.2.0) and links
  both per-language release notes, instead of only python/RELEASE_NOTES.md. The
  hook's INCLUDE_SOURCES map had to follow, or the included links would break
  the now-strict build.
- New scripts/check_python_api_pages.py runs as a gen-files script and fails the
  build if sid.__all__ and docsite/api/python/ disagree in either direction, or
  if a stub is missing from SUMMARY.md. Closes the hole where a newly exported
  function silently misses the site. Verified it catches both a missing page and
  an orphan page.
- Repo URL derived from mkdocs.yml repo_url in both the link-rewrite hook (via
  the config it already receives) and build_matlab_examples.py (via
  mkdocs_gen_files.config, falling back when standalone), instead of hardcoded
  in three places.
- exclude_docs: hooks/ -- mkdocs was copying the hook .py files AND their
  __pycache__/*.pyc into the published site as static assets.

Verified: mkdocs build --strict exits 0 from a clean tree (was 39 warnings), all
11 notebooks execute, both API references generate, and no internal document
(decisions/analyses/plans/DESIGN/REVIEW_CONTEXT/roadmap/todo) appears anywhere
under site/. ruff clean on the changed scripts and hooks.

Refs #110
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant