Skip to content

feat(standing_rules): inject_count + last_injected — named for what it measures - #296

Merged
AVADSA25 merged 1 commit into
mainfrom
feat/standing-rules-inject-count
Jul 24, 2026
Merged

feat(standing_rules): inject_count + last_injected — named for what it measures#296
AVADSA25 merged 1 commit into
mainfrom
feat/standing-rules-inject-count

Conversation

@AVADSA25

Copy link
Copy Markdown
Owner

Requested as fired_count. Shipping it as inject_count, because "fired" is not measurable here — and the requester pre-authorised that fallback if so.

Where a rule is genuinely "applied"

The complete lifecycle:

prompt_block() → string → sys_prompt += block → sent to the LLM

That is the whole mechanism. There is no downstream point where a rule fires. Nothing in the pipeline observes whether the model consulted it, obeyed it, or ignored it.

Measuring real influence would require:

  • ablation — run the turn with and without the rule and diff; doubles cost, and non-determinism makes a single diff weak evidence
  • asking the model which rules it used — precisely the unverifiable self-report codec_claim_check exists to refuse

A field named fired_count would be a fabricated metric wearing an authoritative name, in the module whose entire job is refusing that.

What the number is actually good for

Value What it proves
inject_count == 0 dead weight — the rule never even reached the model. Safe to delete.
high inject_count nothing about usefulness — only that you pay for it on every message

The listing states this in the user-facing text rather than implying the model used anything, and calls out rules that have never reached the model as safe to delete.

Design details

  • record=True is passed only by the live chat path — listing, preview and tests default to False, so the metric cannot measure itself
  • Migration is additive: existing rules keep id/text/added and gain inject_count: 0 / last_injected: null. The store is never reset.
  • Telemetry write failures are caught — a counter must never break a reply

Verification

Migration on a real pre-existing store:

before: {'id': 'old1', 'text': 'Always answer in French.', 'added': '2026-07-01T10:00:00+00:00'}
after : {... 'added': '2026-07-01T10:00:00+00:00', 'inject_count': 0, 'last_injected': None}

Three injections → counts [1,1] → [2,2] → [3,3], persisted to disk with timestamps.

20 standing-rules tests, including "preview does not count itself", "count survives a reload", and one asserting the listing text never uses the word "fired".

…t measures

Deliberately NOT fired_count. A standing rule's entire lifecycle is
prompt_block() -> string -> appended to sys_prompt -> sent to the LLM. Nothing
downstream reports whether the model consulted the rule, obeyed it, or ignored
it. Injection is the only observable event in the pipeline.

Measuring real influence would need ablation (run the turn with and without the
rule and diff — doubles cost, and non-determinism makes one diff weak evidence)
or asking the model which rules it used, which is precisely the unverifiable
self-report codec_claim_check exists to refuse. A field called fired_count would
be a fabricated metric wearing an authoritative name, in the module whose whole
job is refusing that.

What the number is good for: inject_count == 0 PROVES dead weight — the rule
never even reached the model. A high count proves nothing about usefulness, only
that you're paying for the rule on every message. The listing says exactly this
rather than implying the model used anything.

record=True is passed ONLY by the live chat path; listing, preview and tests
default to False so the metric can't measure itself.

Migration is additive: existing rules keep id/text/added and gain inject_count 0
/ last_injected null. The store is never reset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AVADSA25
AVADSA25 merged commit 584ddc6 into main Jul 24, 2026
1 check passed
@AVADSA25
AVADSA25 deleted the feat/standing-rules-inject-count branch July 24, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants