feat(observability): record per-skill drop reasons in skills_filter step - #1553
Open
azzzxxxh1999-bit wants to merge 1 commit into
Open
feat(observability): record per-skill drop reasons in skills_filter step#1553azzzxxxh1999-bit wants to merge 1 commit into
azzzxxxh1999-bit wants to merge 1 commit into
Conversation
`filtered_skill_ids` records which skills survived the skills_filter step but not why any were dropped (TokenRhythm#54). Add a companion `filtered_skill_reasons` mapping (skill ID -> stable reason code) covering all three drop surfaces: - deterministic gate: `disable_model_invocation`, `eligibility_failed`, `missing_required_tools`, `superseded_by_toolset` - meta-skill visibility: `meta_hidden_auto_trigger_off` - retrieval: `not_in_top_k`, plus `retrieval_failed` when the retriever fails outright with candidates available Additive-only changes: - new `ctx.metadata["filtered_skill_reasons"]` key (best-effort, same try/except guard as the existing survivor list) - new optional `drop_reasons` parameter on `_deterministic_gate` (existing call sites unchanged) - new `filtered_skill_reasons` field on `PipelineStepRecord`; readers of older decision-log rows hydrate it as `None` via `_filter_payload` - `dropped_skills` added to the `skills_filter.applied` debug log No changes to the v4 wire protocol, gating behavior, retrieval ranking, or injection. Ruff clean; mypy clean on changed files; the engine/skills/observability suites pass (1995 tests), and a new `tests/test_observability/test_filtered_skill_reasons.py` covers the reason codes, JSONL round-trip, and backward-tolerant reads. Fixes TokenRhythm#54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
filter_skillscurrently publishesfiltered_skill_ids— the list of skillsthat survived the step — but record nothing about the skills that were
dropped, so operators cannot answer "why did skill X not surface this turn?"
(#54). This PR adds a companion
filtered_skill_reasonsmapping(skill ID -> stable reason code) covering all three drop surfaces:
_deterministic_gate)disable_model_invocation,eligibility_failed,missing_required_tools,superseded_by_toolsetmeta_hidden_auto_trigger_offHybridRetrievertop-k selection)not_in_top_k;retrieval_failedwhen the retriever returns an empty result while candidates were available (full-failure paths, incl. zero lexical hits)Reason codes are static tokens, consistent with the existing
fallback_reason/vision_followup_gate_reason_codestyle, so they are safefor the default decision log.
Wire / behavior compatibility
ctx.metadata["filtered_skill_reasons"]key, published under the samebest-effort try/except guard as the existing survivor list.
drop_reasonsparameter on_deterministic_gate; existing callsites and semantics are unchanged (default
Nonerecords nothing).filtered_skill_reasons: dict[str, str] | None = Nonefield onPipelineStepRecord;run_pipelinewires it forfilter_skills(andNonefor every other step). Readers of pre-existing decision-log rows hydrate the
new field as
Nonevia_filter_payload(additive-only, no SCHEMA_VERSIONbump).
dropped_skillsadded to theskills_filter.applieddebug log.memory_onlyearly-return path publishes an empty mapping to keep themetadata contract consistent.
injection.
Tests
tests/test_observability/test_filtered_skill_reasons.py(9 cases):gate reason codes per condition,
not_in_top_kend-to-end with a realloader + lexical retriever,
retrieval_failedshape, decision-log JSONLround-trip, backward-tolerant read of a pre-[Feature]: skills_filter step is opaque — filtered_skill_ids records which skills survived but not why any were dropped #54 row (hydrates as
None),and
run_pipelinewiring.tests/test_skills/test_skill_disable_toggle.py: gate reason-code coverageplus an optional-parameter regression case.
tests/test_skills_default_prompt_contract.py: lexical filter contract nowalso asserts the drop reason map.
Verification (Windows, Python 3.12):
uv run pytestontests/test_observability/,tests/test_skills/,tests/test_skills_default_prompt_contract.py,tests/test_engine/test_runtime_meta_invoke_surfacing.py,tests/test_engine/test_goal_routing_hint.py: 1995 passed, 14 skipped(pre-existing GBK-migration setup errors on Windows are unrelated and
reproduce on unmodified
main).ruff checkon all changed files: clean.mypyon the three changed source files: no new errors.Linked issue
Fixes #54
Release Note
Release note: Decision-log
filter_skillsrecords now include afiltered_skill_reasonsmap explaining why each non-surviving skill wasdropped (gate, meta-visibility, or retrieval).
Safety
No secrets, private transcripts, docs files, test profiles, screenshots, local
packages, or other local-only artifacts are included. No configuration or
persistence-format changes beyond the additive decision-log field.
Third-Party Origin
Third-party origin: none
Documentation Changes
No docs changes included.