Skip to content

feat(observability): record per-skill drop reasons in skills_filter step - #1553

Open
azzzxxxh1999-bit wants to merge 1 commit into
TokenRhythm:mainfrom
azzzxxxh1999-bit:issue-54-skill-drop-reasons
Open

feat(observability): record per-skill drop reasons in skills_filter step#1553
azzzxxxh1999-bit wants to merge 1 commit into
TokenRhythm:mainfrom
azzzxxxh1999-bit:issue-54-skill-drop-reasons

Conversation

@azzzxxxh1999-bit

Copy link
Copy Markdown

Scope

filter_skills currently publishes filtered_skill_ids — the list of skills
that survived the step — but record nothing about the skills that were
dropped, so operators cannot answer "why did skill X not surface this turn?"
(#54). This PR adds a companion filtered_skill_reasons mapping
(skill ID -> stable reason code) covering all three drop surfaces:

Drop surface Reason codes
Deterministic gate (_deterministic_gate) disable_model_invocation, eligibility_failed, missing_required_tools, superseded_by_toolset
Meta-skill visibility (auto-trigger off) meta_hidden_auto_trigger_off
Retrieval (HybridRetriever top-k selection) not_in_top_k; retrieval_failed when the retriever returns an empty result while candidates were available (full-failure paths, incl. zero lexical hits)

Reason codes are static tokens, consistent with the existing
fallback_reason / vision_followup_gate_reason_code style, so they are safe
for the default decision log.

Wire / behavior compatibility

  • New ctx.metadata["filtered_skill_reasons"] key, published under the same
    best-effort try/except guard as the existing survivor list.
  • New optional drop_reasons parameter on _deterministic_gate; existing call
    sites and semantics are unchanged (default None records nothing).
  • New filtered_skill_reasons: dict[str, str] | None = None field on
    PipelineStepRecord; run_pipeline wires it for filter_skills (and None
    for every other step). Readers of pre-existing decision-log rows hydrate the
    new field as None via _filter_payload (additive-only, no SCHEMA_VERSION
    bump).
  • dropped_skills added to the skills_filter.applied debug log.
  • memory_only early-return path publishes an empty mapping to keep the
    metadata contract consistent.
  • No changes to the v4 wire protocol, gating behavior, retrieval ranking, or
    injection.

Tests

  • New tests/test_observability/test_filtered_skill_reasons.py (9 cases):
    gate reason codes per condition, not_in_top_k end-to-end with a real
    loader + lexical retriever, retrieval_failed shape, decision-log JSONL
    round-trip, backward-tolerant read of a pre-[Feature]: skills_filter step is opaque — filtered_skill_ids records which skills survived but not why any were dropped #54 row (hydrates as None),
    and run_pipeline wiring.
  • tests/test_skills/test_skill_disable_toggle.py: gate reason-code coverage
    plus an optional-parameter regression case.
  • tests/test_skills_default_prompt_contract.py: lexical filter contract now
    also asserts the drop reason map.

Verification (Windows, Python 3.12):

  • uv run pytest on tests/test_observability/, tests/test_skills/,
    tests/test_skills_default_prompt_contract.py,
    tests/test_engine/test_runtime_meta_invoke_surfacing.py,
    tests/test_engine/test_goal_routing_hint.py: 1995 passed, 14 skipped
    (pre-existing GBK-migration setup errors on Windows are unrelated and
    reproduce on unmodified main).
  • ruff check on all changed files: clean.
  • mypy on the three changed source files: no new errors.

Linked issue

Fixes #54

Release Note

Release note: Decision-log filter_skills records now include a
filtered_skill_reasons map explaining why each non-surviving skill was
dropped (gate, meta-visibility, or retrieval).

Safety

No secrets, private transcripts, docs files, test profiles, screenshots, local
packages, or other local-only artifacts are included. No configuration or
persistence-format changes beyond the additive decision-log field.

Third-Party Origin

Third-party origin: none

Documentation Changes

No docs changes included.

`filtered_skill_ids` records which skills survived the skills_filter
step but not why any were dropped (TokenRhythm#54). Add a companion
`filtered_skill_reasons` mapping (skill ID -> stable reason code)
covering all three drop surfaces:

- deterministic gate: `disable_model_invocation`,
  `eligibility_failed`, `missing_required_tools`,
  `superseded_by_toolset`
- meta-skill visibility: `meta_hidden_auto_trigger_off`
- retrieval: `not_in_top_k`, plus `retrieval_failed` when the
  retriever fails outright with candidates available

Additive-only changes:

- new `ctx.metadata["filtered_skill_reasons"]` key (best-effort, same
  try/except guard as the existing survivor list)
- new optional `drop_reasons` parameter on `_deterministic_gate`
  (existing call sites unchanged)
- new `filtered_skill_reasons` field on `PipelineStepRecord`; readers
  of older decision-log rows hydrate it as `None` via `_filter_payload`
- `dropped_skills` added to the `skills_filter.applied` debug log

No changes to the v4 wire protocol, gating behavior, retrieval
ranking, or injection. Ruff clean; mypy clean on changed files; the
engine/skills/observability suites pass (1995 tests), and a new
`tests/test_observability/test_filtered_skill_reasons.py` covers the
reason codes, JSONL round-trip, and backward-tolerant reads.

Fixes TokenRhythm#54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: skills_filter step is opaque — filtered_skill_ids records which skills survived but not why any were dropped

1 participant