Add sharding support to HACS data generation pipeline by ludeeus · Pull Request #5419 · hacs/integration

ludeeus · 2026-07-23T05:47:30Z

Summary

Adds sharding to the HACS data generation pipeline. Large categories are split into independent shards that are generated separately and merged back into a single per-category result before summary/validation/publish. Motivated by the integration category, whose single job had grown too large; splitting it keeps each job within per-job time/rate limits and makes runs independently retriable.

The generation matrix runs serially (max-parallel: 1, retained from #5416) to stay within GitHub API rate limits on the shared token — so sharding here provides smaller, resilient, independently-mergeable jobs rather than wall-clock speedup.

Key changes

Sharding: shard_for() (stable, lowercased MD5 hashing) assigns repositories to shards; _slice_by_shard() partitions by that assignment. Disjoint, and the union equals the unsharded set, so merged output matches an unsharded run.
CLI: argparse with --shard x/y (1-based). Omitting --shard produces full, unsharded output (unchanged; used by tests and the single-repository validate.yml path); passing any shard — including 1/1 — writes a partial, so every sharded invocation flows through the merge step uniformly.
Finalization: summary, validation and diff writing moved into finalize_category_output(), run once on the merged dataset. Sharded runs write partial data.json + stored.json under outputdata/_shards/, recombined by the merge step.
New merge script scripts/data/merge_category_data.py.
Deterministic output: repositories.json sorted explicitly (sort_keys is a no-op on a list); data.json/stored.json keys sorted — published files are stable regardless of shard-assembly order.
Workflow: generate-matrix computes shard counts (integration=3, plugin=2, others=1); category-data runs per shard, serially; new merge job recombines a category's shards and uploads the per-category artifact in the original shape; summarize depends on merge and skips directories without a summary.json.

Tests

Unit tests: shard assignment stability/case-insensitivity, --shard x/y parsing, slice disjointness/completeness.
Network-free roundtrip test: write_shard_output + the merge loader reassemble the full set across 1/2/3 shards.
Existing snapshot tests unchanged (unsharded path is byte-for-byte identical).

https://claude.ai/code/session_01NL8ieP9vEVJLTZ51esR6VD

Copilot

Pull request overview

Adds deterministic sharding to the HACS data generation pipeline so large categories can be generated in parallel and then merged into a single validated output, with CI updated to orchestrate shard runs + merging.

Changes:

Added stable shard assignment (shard_for) and shard slicing to generate per-shard partial outputs under outputdata/_shards/.
Extracted finalization (summary/validation/diff writing) into finalize_category_output() and introduced a new merge_category_data script to finalize merged shard outputs.
Updated GitHub Actions workflow to generate shards via a matrix, download shard artifacts, merge them, then summarize/publish.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`scripts/data/generate_category_data.py`	Adds shard hashing/slicing, shard arg parsing, shard output writing, and a shared finalization function.
`scripts/data/merge_category_data.py`	New script that loads shard partials and runs the shared finalization step on the merged dataset.
`.github/workflows/generate-hacs-data.yml`	Refactors the workflow to run per-shard generation, merge shard artifacts, then summarize/publish.
`tests/scripts/data/test_generate_category_data.py`	Adds unit + integration-style tests for shard assignment, parsing, slicing, and sharded-vs-unsharded equivalence.
`tests/conftest.py`	Extends HTTP session proxying to cover the new merge script.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The generate-hacs-data workflow processed each category in a single job. The integration category is by far the largest, so that job dominated runtime and was the most likely to hit GitHub rate limits or time out. Split generation into shards (integration ×3, plugin ×2, everything else ×1) and merge each category's shards back into a single result before the existing summarize/publish jobs run. - generate_category_data: add a stable, salt-free `shard_for` helper and thread a shard `index/total` through generation, slicing both the published data and the default-file repo list by the same lowercased full_name so shards are disjoint and their union equals the unsharded set. Refactor the summary/diff/validation/writing tail into a reusable `finalize_category_output`; a `--shard x/y` CLI selects a shard and writes a partial artifact. With `--shard 1/1` (the default) behaviour is unchanged. - merge_category_data: new script that concatenates the shard partials and runs finalize once on the full merged dataset. - workflow: emit a shard matrix, run generation per shard, add a merge job per category that rebuilds the per-category artifact in the same shape as before, and repoint summarize/publish accordingly. - tests: cover shard_for/_parse_shard/_slice_by_shard and assert a sharded generate + merge matches the unsharded output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8ieP9vEVJLTZ51esR6VD

- generate_category_data: distinguish a sharded invocation from an unsharded one by whether a shard was passed, not by shards > 1. With --shard omitted the full category output is produced (unchanged); with any shard given -- including 1/1 -- a partial artifact is written, so the single-shard categories go through the merge step uniformly like the workflow expects (previously they took the finalize path and the shard artifact upload had nothing to upload). - finalize_category_output: sort repositories.json explicitly (sort_keys is a no-op on a list) and sort the keys of data.json/stored.json, so the published output is deterministic regardless of shard-assembly order. - tests: replace the network-driven sharded/unsharded equivalence test (which required API-usage snapshots) with a network-free roundtrip test of write_shard_output + _load_shard_files across 1/2/3 shards; drop the now-unused merge ClientSession patch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NL8ieP9vEVJLTZ51esR6VD

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

    strategy:
      fail-fast: false
      max-parallel: 1
      matrix:
-        category: ${{ fromJSON( needs.generate-matrix.outputs.categories )}}
+        include: ${{ fromJSON( needs.generate-matrix.outputs.shards )}}


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ludeeus · 2026-07-23T17:04:44Z

Closing in favor of splitting this into smaller, independently-reviewable PRs (the diff here got large). The work will land as a sequence:

Refactor: extract finalize_category_output (no behavior change) — Refactor generate_category_data: extract finalize_category_output #5422
Deterministic output ordering
Sharding capability in the generator (workflow unchanged)
Merge script
Activate sharding in the workflow

Each merges safely on its own; only the last changes the running workflow. Opening the next PR after each merges. Branch left intact for reference.

Generated by Claude Code

Copilot AI review requested due to automatic review settings July 23, 2026 05:47

ludeeus added the pr: action Changes to actions label Jul 23, 2026

Copilot started reviewing on behalf of ludeeus July 23, 2026 05:48 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread .github/workflows/generate-hacs-data.yml

Comment thread scripts/data/generate_category_data.py

claude added 2 commits July 23, 2026 06:18

ludeeus force-pushed the claude/generate-data-workflow-sharding-riz4o4 branch from b61c4b3 to ba5fbb6 Compare July 23, 2026 06:19

ludeeus requested a review from Copilot July 23, 2026 06:22

Copilot started reviewing on behalf of ludeeus July 23, 2026 06:23 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Potential fix for pull request finding

0ba849d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ludeeus mentioned this pull request Jul 23, 2026

Refactor generate_category_data: extract finalize_category_output #5422

Open

ludeeus closed this Jul 23, 2026

github-actions Bot locked and limited conversation to collaborators Jul 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add sharding support to HACS data generation pipeline#5419

Add sharding support to HACS data generation pipeline#5419
ludeeus wants to merge 3 commits into
mainfrom
claude/generate-data-workflow-sharding-riz4o4

ludeeus commented Jul 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

ludeeus commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

ludeeus commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

ludeeus commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ludeeus commented Jul 23, 2026 •

edited

Loading