Skip to content

Add daily moderation quality statistics with late-event repair - #538

Merged
Szer merged 2 commits into
mainfrom
feat/vahter-quality-metrics
Sep 30, 2026
Merged

Szer merged 2 commits into
mainfrom
feat/vahter-quality-metrics

Conversation

@Szer

@Szer Szer commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Persist correction-based TP/TN/FP/FN counts for fast dashboard queries, separating actual pipeline totals, counterfactual ML, and matched ML/LLM comparisons by recorded deployment. Keep abstained, unresolved, and excluded cases visible.

The scheduled job atomically replaces today and the previous 14 UTC days, plus older days affected by edits, reviewer corrections, or unbans. An event-insert trigger queues work in the event transaction; resolving pending IDs into durable dirty days and acknowledging those IDs is atomic. This handles out-of-order commits without a high-water event-ID checkpoint, and failed rebuilds retain their dirty markers. All verdict interpretation remains in F#.

Authenticated POST /quality-history runs the daily refresh; ?day=YYYY-MM-DD rebuilds one day. Explicit ?mode=backfill fills older missing days in resumable batches (default seven), separately from the daily job. Includes service-role grants, dashboard SQL with provisional/freshness fields, and structured runtime logging with a warning above the 30-second daily budget.

Validation: Release build and nine PostgreSQL integration tests passed, covering corrected counts, reversal, unban, old edits, today's provisional counts, backfill boundaries, late commits with lower IDs, rollback, failed-rebuild retry, and endpoint authentication. Mandatory repository hooks passed.

Local synthetic benchmark using the actual job: 20,148 messages / 49,006 events across 15 days; daily refresh with 2,729 pending events took 1.15–1.45 seconds, and processing the full 49,006-event backlog took 1.71 seconds. Repeated aggregates matched the evaluator. These are local measurements; production end-to-end runtime remains to be verified after deployment.

@Szer Szer changed the title Add historical ML and LLM moderation quality statistics Add daily moderation quality statistics with late-event repair Sep 30, 2026
@Szer
Szer merged commit 5e7ec91 into main Sep 30, 2026
8 checks passed
@Szer
Szer deleted the feat/vahter-quality-metrics branch September 30, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant