Add phase 35: crawl the reference tree the TOC never exposes - #26
Add phase 35: crawl the reference tree the TOC never exposes#26pedropaulovc wants to merge 7 commits into
Conversation
Phase 35 needs the same identity rules phase 115 already worked out -- canonical_key, the /api boundary, reference-page detection, link iteration. Lift them out of link_targets.py so both crawls agree on what counts as the same page; link_targets keeps only its seed/boundary logic. No behaviour change: phase 115's tests pass untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 10/30 save pages under a hashed name (Type_members_<hash>.html); the reference tree that phase 35 crawls uses the bare upstream name (Type_members.html), and spells the assembly both SolidWorks.Interop.* and SOLIDWORKS.Interop.*. - is_type_file matched "_members_" as an infix only, so a bare Type_members.html read as a type page. - extract_namespace_from_filename passed the assembly and namespace through verbatim, so the two spellings became two distinct assemblies. Phase 90 then writes SOLIDWORKS.Interop.sldworks.xml alongside the real SolidWorks.Interop.sldworks.xml. - Page titles were reduced by chained str.replace of " Interface"/ " Class"/" Enumeration"; the reference tree's shape is "X Delegate (SolidWorks.Interop.sldworks)". strip_page_title_kind handles both, and leaves an unrecognised title intact rather than truncating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phases 10 and 30 discover types from the Table of Contents, so a type that no TOC node points at is invisible to the whole pipeline no matter how many pages link to it. IBody, IFace, IModelDoc and 41 others sit in that hole, as do 36 enumerations -- including swDimXpertGtolType_e, which the source TOC drops because its "Enumerations" nodes for ids 2.3/2.4/2.6/2.7 return the Interfaces list verbatim. Phase 35 seeds from the ~-style reference pages the corpus already links to, fetches each one's _members companion, and expands only those -- every other reference page is a leaf, so the crawl cannot walk off into the full reference tree. Assemblies outside the shipped set are recorded and dropped (24 pages, mostly routingapi and swdocmgrapi). Phases 20/40/50/60 take --input-dirs (plural) so they read phase 35's HTML alongside the TOC crawl's. Crawl: 887 seeds -> 748 pages, 0 failures, 213 members discovered. Against the released corpus, purely additive -- nothing dropped: phase 20 553 -> 597 types phase 40 1529 -> 1955 types phase 50 12914 -> 13192 members phase 60 963 -> 999 enums (all 36 new ones have members) phase 90 12 assemblies, unchanged; every statistic up All 1594 members of the 44 new types resolve to a detail page, so none of them ship as name-and-signature stubs. Pipeline metadata and the release manifests are deliberately untouched; regenerating them is the next release cut's job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95c420e2a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
run_pipeline.sh
- The post-phase-35 phase-20 refresh was numbered 20, and `run` skips any
phase below --from, so `--from 35` -- the documented way to resume a
failed reference crawl -- silently dropped the refresh that resume
exists to produce. Numbered 36.
- Phase 35 also lists phases 70/100/115 as seed sources but ran before
all three, so a reference page linked only from an example, the
programming guide, or a phase-115 page never got seeded. Added a
--resume second pass after 115 plus the local re-extractions that fold
it into the exports. Measured cost on the 2026 corpus: 4 extra pages
(seed 525 -> 529).
35_crawl_referenced_types
- run_crawler exited 0 no matter how many requests died, and the phase has
no validation step, so a total outage would have published a partial
reference set as if it were complete. crawl_failure() now decides;
it lives in reference_targets so tests reach it without importing
scrapy (every crawl phase ships a package named solidworks_scraper, so
a plain import resolves to whichever phase hit sys.path first).
- The manifest claimed robots.txt was respected with a 2s delay while the
settings say False/0.1s -- a false compliance record. It is now read
from the settings in force, and the committed manifest regenerated.
60_extract_enum_members
- Reduced titles with `.replace(" Enumeration", "")`. No enum page in the
corpus carries the reference tree's parenthetical suffix (checked all
36: every one is plain "<Name> Enumeration"), so this was not the live
duplicate-type bug the review described -- but the substring replace is
the fragile form regardless. Uses strip_page_title_kind now; output is
byte-identical on all 999 enums.
READMEs
- Phases 20/40/50 still documented --input-dir. Updated to --input-dirs
rather than adding an alias: this project does not carry backwards
compatibility, and a stale doc is the actual defect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60bf33a4bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All four are real; all four confirmed against the code.
- A failed HTML write was swallowed: the item continued to
MetadataLogPipeline (400, after HtmlSavePipeline's 300), which recorded
the URL in urls_crawled.jsonl with no file behind it. The page was then
missing from extraction *and* skipped by the next --resume as already
crawled. Now raises DropItem and counts a failure, so run_crawler exits
non-zero and the URL stays eligible for retry.
- crawl_stats.json was read wherever it happened to be: a spider that died
before closed() left a fresh run reading {} (no problem reported) and a
--resume run reading the *previous* run's clean statistics. It is now
unlinked before the crawl and its absence afterwards is a failure.
- MAX_PAGES could not bind. Seeds are all scheduled in start_requests, and
the check sat only in front of expansion, so a member list received just
under the limit still enqueued its whole member set. The cap moved into
schedule(), where every request is created. In-flight requests can still
land after it trips, so the ceiling is MAX_PAGES plus at most one
concurrency window -- documented rather than claimed exact.
- Phases 70/100/115 were scanned on the first pass, but they run later, so
on a repeat refresh they held the *previous* cycle's HTML and could seed
reference types the current corpus no longer links to (and, now that
crawl failures are fatal, a removed page would fail the phase). Pass 1 is
restricted to 10/30; --all-sources opts the rest in for the post-115 pass.
Measured on the 2026 corpus: 525 seeds early, 529 with all sources.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ace3af464a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Four of five applied; the fifth is answered with a measurement. - The seed scan included this phase's own HTML, which bypasses the leaf boundary parse_page enforces (member-list pages expand, everything else is a leaf), so each --resume could walk another step into the reference tree. Its metadata still bounds the crawl; its HTML no longer seeds it. Zero seed difference on the 2026 corpus -- the frontier is already closed -- so this makes the invariant explicit rather than relying on that staying true. - --resume excluded every recorded URL without checking its HTML is still there. metadata/ is committed and output/ is gitignored, so on a fresh checkout the documented resume mode skipped all 748 pages and left nothing to extract. Now uses build_saved_page_keys, the fix PR #24 established for this same bug class; moved from phase 115's link_targets into shared/api_urls since two phases need it. - A truncating page cap reported success: thousands of pages saved makes `crawled` look healthy while the reference set is knowingly incomplete. Non-zero unscheduled_pages is now a failure. - Malformed or unshaped __NEXT_DATA__ counted as a skip, and skips never reach crawl_failure, so a mangled response dropped a type with the phase still green. Those are failures now. The recognised soft-404 (200 with empty helpText) stays skippable -- and that distinction matters: the pass-2 seed is *entirely* soft-404s, so treating them as failures would have made the pipeline fail on the real corpus. crawl_failure no longer calls an all-skipped run a failure either. Not changed: re-running phases 70/80/115 after pass 2. Measured instead -- all 4 pass-2 seeds are soft-404s (checked over HTTP: 200, helpContentData present, helpText empty), so pass 2 adds no page that could reference a new example or /api page. Chasing it properly needs a fixpoint loop, not one more round; the regress is real but empty here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb01128118
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- A failed append to urls_crawled.jsonl was only logged, so the HTML sat on
disk with nothing mapping it back to its URL: source discovery works off
that manifest, so the page was orphaned and the next --resume re-fetched
it, all while the phase reported success. Counts as a failure now, same
as a failed HTML write.
- crawl_failure ignored how the spider closed. scrapy writes a valid stats
file whatever the reason, so a graceful interruption ("shutdown") left a
partial crawl looking clean and the pipeline went on to publish it. Only
reason == "finished" passes.
The second one tightens an existing contract: crawl_failure({}) used to be
"not a failure" and is now a failure, since a spider that died before
closed() leaves exactly that. The test that asserted the old behaviour is
updated rather than worked around.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b256ecd0be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sources = [] | ||
| for phase in others: | ||
| sources += crawled_html_sources(REPO_ROOT / phase, _metadata(phase)) |
There was a problem hiding this comment.
When a crawl is resumed after a member request fails, is interrupted, or hits the page cap, its already-saved _members page is added to crawled_keys, while this source list excludes phase 35's HTML. Since parse_page() is the only place that expands a member list, the missing child is never scheduled again; the resumed run can then exit successfully with an incomplete type. Preserve the leaf boundary by selectively replaying saved _members pages or otherwise persisting their pending requests rather than omitting the entire derived frontier.
Useful? React with 👍 / 👎.
| if assembly not in shipped_assemblies: | ||
| out_of_scope[assembly or "?"] = out_of_scope.get(assembly or "?", 0) + 1 | ||
| continue |
There was a problem hiding this comment.
When the same external-tree page is linked more than once, this increments the assembly count for every occurrence because out-of-scope keys are never added to a deduplication set. An absolute href is normally yielded twice by iter_page_links()—once by the absolute-URL regex and once by the attribute scanner—so even a single link inflates out_of_scope_pages; repeated links across the corpus make the recorded metric reflect link frequency rather than the distinct skipped-page surface that the manifest and docstring claim to monitor.
Useful? React with 👍 / 👎.
1 participant
The TOC is the only way types enter this pipeline, so a type no TOC node points at is invisible to every phase downstream — however many pages link to it.
IBody,IFace,IModelDocand 41 others sit in that hole. So do 36 enumerations, includingswDimXpertGtolType_e: the source TOC's "Enumerations" nodes for ids 2.3/2.4/2.6/2.7 return the Interfaces list verbatim, which is a defect upstream, not here.Phase 35 seeds from the
~-style reference pages the corpus already links to, fetches each one's_memberscompanion, and expands only those. Every other reference page is a leaf, so the crawl can't walk off into the full reference tree. Assemblies outside the shipped set are recorded and dropped (24 pages, mostlyroutingapiandswdocmgrapi).Crawl: 887 seeds → 748 pages, 0 failures.
Against the released v3.11.0 corpus
Purely additive — no baseline type, member or enum is dropped:
All 1594 members of the 44 new types resolve to a detail page, so none ship as name-and-signature stubs. All 36 new enums have members.
The middle commit is the interesting one
The reference tree uses a different filename dialect than the TOC crawl, and the parsers quietly mangled it. Feeding phase 35's pages in unchanged produced:
three flavours of garbage
is_type_filematched_members_as an infix only. Phase 10 savesType_members_<hash>.html; phase 35 saves the bare upstreamType_members.html, which read as a type page —IBody Memberssitting next to the realIBody._members_alone, leaving the suffix glued on: namespaceSolidWorks.Interop.sldworks.IBody_members. Every member of those 44 types then failed to join with its detail page and would have shipped as a stub.str.replaceof" Interface"/" Class"/" Enumeration", which doesn't know the reference tree'sX Delegate (Namespace)shape.Separately: the site spells the assembly both
SolidWorks.Interop.sldworksandSOLIDWORKS.Interop.sldworks. Unfolded, phase 90 writes a secondSOLIDWORKS.Interop.sldworks.xmlnext to the real one — I hit exactly that while cutting v3.11.0 and had to delete the strays by hand.Both pre-existing parsers are covered by regression tests that fail without the fix (verified by stashing it).
Not included
Pipeline metadata and release manifests are untouched — regenerating them is the next release cut's job, so this PR's diff stays reviewable.