Copy this directory into a GitHub repository, enable Actions with repository write permissions, and run Crawl public PCB links once manually. The PCB workflow then runs every five minutes. No personal access token is needed. Run locally with GITHUB_TOKEN=... python crawler.py --max-repos 10 --max-requests 35.
Outputs under data/: github_links.txt (one immutable GitHub file URL per line), pcb_links.jsonl (repo, format, path, commit, blob SHA), and state.json (cursor, pending page, scan counts, skipped repositories). Commit all three files; the workflow does so automatically. A candidate is validated by downloading its blob and checking its signature. Current coverage: KiCad boards, Altium PcbDoc compound files, Eagle XML boards, LibrePCB .lpp and gEDA .pcb. It indexes board designs, not schematic-only files. Binary validation confirms container type, not full semantic parse; this is a link index, not a complete EDA parser.
Scale and completeness: Enumeration starts at repository ID zero and progresses in creation order. At 10 repositories per run, twelve runs hourly scan at most 120 repos/hour; a full historic crawl is impractical on this schedule. Each repository needs several API calls. Increase the budget only with appropriate API capacity, or use an external persistent worker and storage for a large backfill. Results cover current default-branch snapshots only. Repositories deleted, made private, updated after their turn, or containing unusual PCB formats are absent until separately revisited. skipped records errors for later retry. Links may stop working if a repository is removed or access changes. GitHub Actions scheduled runs can be delayed; GitHub may disable them in inactive public repositories.
Important: The earlier suggestion to scan every public repository using its tree is costly. The exact public-repository enumeration endpoint exists, but this implementation does not promise exhaustive results within a practical timeframe. No source file content is saved, only metadata and links.
boardrepo.py reads public project pages listed in https://boardrepo.com/sitemap.xml, extracts each page’s labeled GitHub source repository, and writes deduplicated GitHub repository URLs to data/boardrepo_links.txt. data/boardrepo_state.json tracks resolved and unresolved pages. Up to 200 new pages are checked per scheduled run, so the initial backfill takes multiple runs. The site advertises public crawling in robots.txt. This is an index of links shown by BoardRepo, not downloaded designs or a validation of native PCB files. GitHub source links, where available, remain in the existing GitHub index. Run locally with python boardrepo.py. The PCB, CAD, and BoardRepo workflows request a run every five minutes. They use separate run locks and retry rebasing and pushing when another workflow commits first. GitHub may delay or skip scheduled runs; 30 runs is the minimum for about 6,000 pages at 200 per run; plan on at least 2.5 hours plus scheduling and processing delays.
cad_crawler.py is a separate GitHub public-repository cursor and scheduled workflow. It writes data/cad_links.txt (direct GitHub file URLs), data/cad_links.jsonl (format, size, commit and validation level) and data/cad_state.json. It indexes STL, STEP/STP, GLB, IGES/IGS, Parasolid X_T/X_B, DWG/DXF, SolidWorks SLDPRT/SLDASM, Inventor IPT/IAM and possible AIM, CATIA CATPart/CATProduct/CGR, ambiguous Creo/NX PRT and Creo ASM, Revit RVT/RFA, possible Archicad PLN, Rhino 3DM, and Fusion 360 F3D/F3Z. AutoCAD BAK is included only if its DWG header is verified.
Files with standardized headers (such as GLB, DWG, STEP, Rhino, Fusion) are checked if 2 MB or smaller. Other records are labeled extension_only or ambiguous_extension; indexing is not full CAD parsing. Blobs over 100 MB are skipped, and Git LFS pointers and repository history are not indexed. .prt, .asm, .pln, .aim and generic backups need format-aware parsers for reliable product classification. The PCB and CAD workflows each cap their API calls at 35 per five-minute run to leave room under the shared GitHub token's hourly limit; long-running repository enumeration takes time.
The workflows request another workflow_dispatch run after successfully saving each batch. This works even if GitHub's cron scheduler does not deliver the schedule event. PCB and CAD wait five minutes before dispatching to stay within the shared GitHub API limit; BoardRepo dispatches the next batch only while its sitemap still contains unprocessed pages. GitHub cron remains as a backup. If a run fails before the continuation step, the chain stops and must be restarted from the Actions page; the persisted cursor prevents rescanning completed work. These continuing jobs occupy GitHub-hosted runners while waiting.