Repository navigation
Conversation
A fresh 'make setup' leaves .venv/ untracked; coverage runs leave .coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXDjHW98R63siawNgUCQzc
Full offline suite under pytest-cov (branch mode), N=275 tests / 2140 statements. Strong: deterministic clinical core at ~100%. Gaps, in priority order: llm.py agent_loop untested (57% branch) + a confirmed structured() bad-JSON bug (raises JSONDecodeError, not LLMFailure, so routes 500 instead of failing closed 502); LLM-backed routes untested even stubbed (/mci/handoff, /mci/triage/note, /mci/compliance); content validator rejection branches dead in corpus.py and sim/cases.py; agent tool bodies uncovered in reconcile.py/compliance.py; goldset extraction eval is manual-only (propose opt-in 'live' marker asserting 0 under-triage FNs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXDjHW98R63siawNgUCQzc
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXDjHW98R63siawNgUCQzc
…eded cap note Rebased onto main after the ecosystem lane + repo-wide ruff 0.16 fix landed; dropped the now-superseded 'cap ruff <0.16' commit in the rebase. Re-ran the suite on the merged tree (291 passed, 80% line / 78% combined, llm.py still 57%) — all documented gaps and priorities stand. Relabeled the lane T4 (T3 was taken by the ecosystem terminal). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TXDjHW98R63siawNgUCQzc
bGOATnote
force-pushed
the
claude/test-coverage-analysis-dv8g5o
branch
from
July 31, 2026 05:07
30da403 to
c2192d5
Compare
bGOATnote
changed the title
bGOATnote
added a commit
that referenced
this pull request
Aug 31, 2026
…t 2026-08-24) * fix: close fail-closed gaps (iteration cap, bad JSON, silent empty review) + ignore .env P1-1: .gitignore now excludes .env/.env.* (keeping .env.example), .venv/, *.pem, *.key, .coverage — the quickstart creates .env at the repo root and nothing guarded it (folds in PR #1's .venv/.coverage lines). P1-2: agent_loop treats stop_reason="tool_use" on the final message (the SDK runner exhausted max_iterations with the model still requesting tools) as LLMFailure, and run_compliance_review raises LLMFailure when the agent never called report_findings — a compliance audit can no longer render as a clean pass with zero findings. P1-3: structured() wraps json.loads so malformed JSON raises LLMFailure (the documented contract) instead of escaping as a raw JSONDecodeError / HTTP 500. Tests: fake tool-runner coverage for agent_loop (success, iteration cap, no messages), invalid-JSON structured(), and both compliance fail-closed paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01142EwVwkXFRGnYu7YgHSYL * docs: true up claims — goldset N, test count, CME draft status, dataset NOTICE P1-4: README metrics line now states the run explicitly (single run, 2026-07-18, on 12 of the current 13 goldset cases; rerun pending) and the test count matches the collected suite (297). The goldset rerun itself is deferred to the owner: python -m halo.mci.demo, then commit evals/goldset_<date>.json. P1-5: docs/ECOSYSTEM.md no longer says "MD-signed" — the CME cards are hand-built with physician sign-off pending (all four carry review.status="draft"). P1-6: synthetic-ambient-fhir-25/NOTICE states the dataset was provided by Abridge for hackathon use and that redistribution terms are unconfirmed (not covered by the repo's Apache-2.0). Confirming terms with Abridge is deferred to the owner. Also: scenario count consistency (6 scripted scenarios, README + demo.py). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01142EwVwkXFRGnYu7YgHSYL * chore: readiness polish — SECURITY/CONTRIBUTING, SHA-pinned actions, mypy strict clean - SECURITY.md (disclosure contact, scope, supported versions) and a short CONTRIBUTING.md. - CI actions pinned to commit SHAs (checkout v4.4.0, setup-python v5.6.0). - anthropic bounded to >=1.0,<2 (code depends on 1.x beta surfaces). - Fix the 5 mypy-strict errors in src/halo/mci (env-default Path handling in panel/census, typed _invert helper in triage, extract values dict) — mypy strict now clean on all 36 source files, matching the log's claim. - STATUS.md archived to docs/HACKATHON_LOG.md (hackathon is over); README and CLAUDE.md references updated. - HandoffRequest.incident_date defaults to today (UTC) instead of the hard-coded hackathon date. - edu/drill.py and edu/lookup.py log at warning level before failing closed, so an auth/config outage is distinguishable from a model decline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01142EwVwkXFRGnYu7YgHSYL --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
Closing as superseded: the .gitignore hardening landed on main via PR #2 (2026-08-31), and the repo is entering passive maintenance. If the coverage analysis doc is still wanted, re-propose it from current main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Analysis-only lane (no product code changes) plus the hygiene fix it surfaced:
docs/TEST_COVERAGE.md— full-suite coverage analysis (pytest-cov, branch mode, freshmake setup): 80% line / 77% branch, N=275 tests / 2,140 stmts on 2026-07-30; re-verified post-rebase on merged main (291 tests / 2,232 stmts, 80% line / 78% combined) with all priorities standing:halo/llm.pyis the worst-tested module (57% branch):agent_loop()has zero tests, and probing found a confirmed latent bug —structured()raises rawJSONDecodeError(notLLMFailure) on malformed/empty JSON, so routes 500 instead of failing closed with 502./mci/handoff,/mci/triage/note,/mci/compliance) have no tests, even stubbed.edu/corpus.pyandsim/cases.py(the "corpus decides" safety gate is unverified).@beta_toolbodies inreconcile.py/compliance.pynever execute in tests; the agent's add-a-new-candidate merge path is uncovered.livepytest marker that asserts it..gitignore— add.venv/and.coverage(a freshmake setupleft the venv untracked).STATUS.md— lane row (T4) + decisions-log entry per the multi-terminal protocol.History note
This branch originally also capped
ruff<0.16after hitting the 0.16 default-rule expansion on a fresh setup; the repo-wide fix +ruff==0.16.1pin landed on main first (T3's lint pass), so the cap commit was dropped during rebase — that story is recorded in the STATUS decisions log.Verify
make check— green on merged main (lint clean under ruff 0.16.1, 291 passed). Branch is rebased onmain.🤖 Generated with Claude Code
https://claude.ai/code/session_01TXDjHW98R63siawNgUCQzc