Skip to content

feat(llm-worker): add offline engagement aggregate - #3302

Merged
Xore merged 1 commit into
mainfrom
issue-3293-coder
Sep 25, 2026
Merged

Xore merged 1 commit into
mainfrom
issue-3293-coder

Conversation

@Xore

@Xore Xore commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Summary

  • extend the existing llm-worker owner with a network-free --offline-engagement-aggregate command over the committed labeled corpus
  • add fixed command-count buckets, covered/uncovered session counts, terminal-observation breakdowns, and covered-only captured-close/idle-finalization rates
  • always report the separately counted labeled benign baseline; omit rates when the covered denominator is zero
  • add regression tests for missing coverage, baseline denominators, ambiguity, and the real offline corpus output
  • document that the aggregate is descriptive and cannot attribute a client or termination reason

Metrics

  • all_label_count, all_covered_session_count, and all_uncovered_session_count report the reconstructed session population.
  • session_count, covered_session_count, and uncovered_session_count report the non-benign labeled rate population.
  • command_count_buckets uses 0, 1-4, 5-9, 10-19, and 20+; each bucket reports its denominator, terminal observations, uncovered count, and labeled_benign_baseline.
  • captured_close_rate and idle_finalization_rate are emitted only with a nonzero covered denominator, alongside the labeled benign baseline.

Validation

  • python -m unittest discover -s llm-worker/tests -v — 82 tests, OK
  • python llm-worker/worker.py --selftest — PASS
  • python llm-worker/worker.py --offline-engagement-aggregate — 27 corpus events, 10 Cowrie events, 4 sessions, 0 covered, 4 uncovered, 1 labeled benign session, no rate
  • The aggregate path reads only the committed JSONL corpus and does not construct Elasticsearch, model, or status clients.

Dependency

This branch depends on #3301 (issue #3292) for the exact terminal_observation and capture_coverage field names and constants. It does not compile or work standalone against current main without #3301; the tests and aggregate were validated on this branch with #3301's implementation present.

Closes #3293

@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

research: add an offline session-engagement aggregate with labeled benign coverage

1 participant