Add snapshot data support to category data generation

[ludeeus]

Summary

This PR adds support for using pre-fetched snapshot data during category data generation, enabling faster and more reliable data generation workflows. When snapshot data is available, the generator uses it instead of fetching from the data client.

Key Changes

• New helper functions in generate_category_data.py:

  • get_stored_data(): Returns category data from snapshot directory if available, otherwise fetches from data client
  • get_removed_repositories(): Returns removed repositories list from snapshot directory if available, otherwise fetches from data client
  • Both functions respect the HACS_EXISTING_DATA_DIR environment variable

• GitHub Actions workflow updates (generate-hacs-data.yml):

  • Added new preflight job that fetches and caches existing published data before category generation
  • Updated category-data job to download the cached snapshot data and pass it via HACS_EXISTING_DATA_DIR environment variable
  • Improved matrix generation logic with better bash variable handling and JSON formatting
  • Updated job dependencies to include the new preflight job

• Test coverage in test_generate_category_data.py:

  • Added _StubDataClient and _StubHacs test fixtures for mocking
  • Added tests verifying snapshot data is read from disk when available
  • Added tests verifying fallback to data client fetch when snapshot is unavailable

Implementation Details

• The snapshot data directory is optional and controlled via the HACS_EXISTING_DATA_DIR environment variable
• When the snapshot directory is set, file I/O is used to read pre-fetched data, avoiding redundant network requests
• The implementation maintains backward compatibility - if the environment variable is not set, the original data client fetch behavior is preserved
• The workflow now fetches data for all categories and the removed repositories list in a separate preflight job, making the data available to all subsequent category generation jobs

https://claude.ai/code/session_01L22jYdNGF4hB4fvTYPTzP4

[Copilot]

Pull request overview

This PR adds optional “snapshot” inputs to the category data generator so it can reuse already-published data from disk (when provided) instead of always fetching from the remote data client, and updates the GitHub Actions workflow to prefetch that snapshot once per run.

Changes:

• Add get_stored_data() and get_removed_repositories() helpers to read pre-fetched JSON from HACS_EXISTING_DATA_DIR (with fallback to the data client).
• Update the generator to use the new helpers for stored category data and removed repositories.
• Extend CI with a preflight job to download published data into an artifact, then pass it to category jobs via HACS_EXISTING_DATA_DIR; add tests for the new helper behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
scripts/data/generate_category_data.py	Adds snapshot-aware helper functions and wires them into category generation.
tests/scripts/data/test_generate_category_data.py	Adds tests/stubs verifying snapshot read behavior and fallback fetching.
.github/workflows/generate-hacs-data.yml	Adds a preflight fetch job + artifact handoff and improves matrix JSON handling.

Comments suppressed due to low confidence (2)

scripts/data/generate_category_data.py:80

• When HACS_EXISTING_DATA_DIR is set but removed.json is missing/invalid, this will raise and fail the whole run. To match the intended "use snapshot when available" behavior, fall back to fetching removed repositories on file/JSON errors.

    if EXISTING_DATA_DIR:
        with open(
            os.path.join(EXISTING_DATA_DIR, "removed.json"), encoding="utf-8"
        ) as file:
            return json.load(file)

tests/scripts/data/test_generate_category_data.py:372

• This assertion should match the (repo-id -> repo-data) shape returned by the real data client and the updated test stub, otherwise the test can pass while still diverging from production expectations.

    assert await get_stored_data(hacs, "plugin") == {"fetched": "plugin"}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

scripts/data/generate_category_data.py:92

• Similarly, get_removed_repositories will return whatever JSON is in removed.json if it parses, even if it isn't a list of repository names. Validating that the snapshot is a list (and falling back when it's not) avoids hard-to-debug failures later in the generation flow.

async def get_removed_repositories(hacs: AdjustedHacs) -> list[str]:
    """Return the removed-repositories list from the snapshot dir when available, else fetch."""
    if (removed := _read_snapshot(hacs, "removed.json")) is not None:
        return removed
    return await hacs.data_client.get_repositories("removed")