Skip to content

chore(scripts): add copy_measure_content.py for cross-server measure content transfer - #394

Open
blakenan-bellese wants to merge 1 commit into
mainfrom
chore/copy-measure-content-script
Open

chore(scripts): add copy_measure_content.py for cross-server measure content transfer#394
blakenan-bellese wants to merge 1 commit into
mainfrom
chore/copy-measure-content-script

Conversation

@blakenan-bellese

Copy link
Copy Markdown
Collaborator

Summary

  • Adds scripts/copy_measure_content.py — copies Measure / Library / ValueSet / CodeSystem from one FHIR server to another as batch Bundles of PUT entries, reusing the shape of fhir_client.push_resources (batch not transaction, so the target does not enforce cross-entry referential integrity).
  • Written to load Lenny's connectathon measure content onto an external MCS. Used to move 185 resources (20 CodeSystem, 138 ValueSet, 19 Library, 8 Measure) to the CMS connectathon server.
  • Adds unit tests for the token-expiry and collision logic.

Related issue

None — utility script extracted from operational work.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Infrastructure / CI/CD
  • Chore (deps, tooling, lockfile)

Checklist

  • Tests added or updated — 20 unit tests, though see the CI gap below
  • docs/ updated — N/A, standalone script with a full module docstring
  • No new ADRs needed — an ops script, no architectural commitment
  • Security implications considered — see below

Safety properties

Each of these came from a bug found while building the script, not from foresight:

  • Read-only collision check before any write. Aborts (exit 3) without --overwrite when the target holds different content at the same IDs, printing each conflict with both versions.
  • Only 404/410 counts as "absent". The first version treated any error as "not there", so an expired token made an unreadable server look like an empty one — it reported 185 new, 0 collisions off the back of 401s, which is a green light to overwrite. Now 401/403/5xx raises.
  • Preflight does one authenticated search before real work. /metadata on the CMS server is unsecured and returns 200 regardless, so it proves nothing about the token.
  • Verification is by direct read, not search count. HAPI's index refresh is async; a measured 40-second lag on that server made a successful write report SHORT and exit 1. Direct reads decide pass/fail now; the search count is still shown, labelled as informational when it lags. Consistent with the async-indexing triage rule in CLAUDE.md.

Security implications

  • Secrets are never command-line arguments. argv is world-readable via ps and lands in shell history. The curl -u CLIENT_ID:CLIENT_SECRET form this replaces leaks both ways.
  • Access token resolution: $TARGET_FHIR_TOKEN0600 cache file → OAuth2 client_credentials grant. Client credentials come from env vars or the macOS keychain.
  • The token cache is opened with O_CREAT at 0600 so it is never briefly world-readable.
  • Expired JWTs are skipped by reading the exp claim (no signature verification — that is the server's job; we only decide whether it is worth sending). A 401 mid-run triggers exactly one refresh-and-retry.
  • The script only ever writes the four content resource types it reads. It does not touch patient data.

Test plan

python3 -m pytest scripts/tests/test_copy_measure_content.py -v    # 20 passed
cd backend && ruff check ../scripts/copy_measure_content.py

Known gap: CI's unit job runs cd backend && pytest tests/, so it does not collect scripts/tests/*.py. These tests run only when invoked manually. Wiring script tests into CI is out of scope here; flagging rather than leaving it implicit.

Dry-run and check-only modes write nothing and are safe to run against any target:

python3 scripts/copy_measure_content.py --target https://example.org/fhir --check-only
python3 scripts/copy_measure_content.py --target https://example.org/fhir --dry-run

Verified end to end against the CMS connectathon server: 185/185 written in 24.6s, all 185 confirmed by direct read (--verify-all), and every Measure.library canonical resolved to a Library present on the target.

🤖 Generated with Claude Code

…content transfer

Copies Measure / Library / ValueSet / CodeSystem from one FHIR server to
another as batch Bundles of PUT entries, reusing the shape of
fhir_client.push_resources (batch not transaction, so the target does not
enforce cross-entry referential integrity).

Written to load Lenny's connectathon measure content onto an external MCS.
Used to move 185 resources to the CMS connectathon server.

Safety properties, each of which came from a bug found while building it:

- A read-only collision check runs before any write and aborts without
  --overwrite when the target holds different content at the same IDs.
- Only 404/410 counts as "absent". A first version treated any error as
  "not there", so an expired token made an unreadable server look like an
  empty one and reported "185 new, 0 collisions" — a green light to
  overwrite based on 401s.
- A preflight does one authenticated search before real work, because
  /metadata is frequently unsecured and proves nothing about the token.
- Verification is by direct read, not search count. HAPI's index refresh is
  async; a measured 40s lag on the CMS server made a successful write look
  like a failure. Search count is still reported, labelled as informational.

Secrets are never command-line arguments (argv is world-readable via ps and
lands in shell history). The access token comes from $TARGET_FHIR_TOKEN, a
0600 cache file, or an OAuth2 client_credentials grant; client credentials
come from env vars or the macOS keychain. Expired JWTs are skipped by `exp`,
and a 401 mid-run triggers one refresh-and-retry.

Tests cover the token-expiry and collision logic. CI's unit job runs
`cd backend && pytest tests/`, so it does not collect scripts/tests/*.py —
run them manually when touching the script.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant