Gc/topic scout improvement#3
Merged
Merged
Conversation
Lead with positioning: Topic Scout is the reusable engine generalized from EvaPaper. Add a "Why not just use Microsoft Scout?" section explaining the core differentiator — a consistent, designed output style (HTML dashboard + Markdown notes) and comparable per-topic analytics that ship for free, so users spend tokens on research judgment rather than reformatting each run. Document the house style, agent-agnostic runtime, ownable git-native corpus, and a template-gallery roadmap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
Replace the hedged "general scouting assistants" framing with the actual product: Microsoft Scout (June 2026, OpenClaw-based) is an always-on Microsoft 365 work agent, not a research-paper tool — no fixed research output format, no persistent topic corpus, no per-topic analytics. Sharpen the comparison table and bottom line accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
build_dashboard.py now reads an optional `theme` block from topic.json
and renders the dashboard's palette, typography, and category colors
from it. Every key is optional and deep-merges over the defaults, so a
partial theme (or none) is always safe and the default broadsheet look
is unchanged.
- resolve_theme()/root_css() merge palette + fonts over DEFAULT_THEME
and emit a :root{} custom-property block injected via __ROOT__.
- Convert hardcoded fonts (display/body), accent colors, and the
editorial surfaces (panels, metric cells, bar tracks, wiki nav,
buttons, table + chart gridlines) to CSS variables so palette
overrides retint the whole layout — enabling coherent dark themes.
- The citation-graph module stays a fixed dark inset by design.
- category_colors drive taxonomy bars, the trend chart, and the graph.
- Document the theme block in README + topic.example.json.
- Tests: resolve_theme merge/fallback + end-to-end themed render.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
Replace the single opaque 0-10 LLM score with a transparent judging pipeline, tunable from a `judging` block in topic.json. - Blind the judge from the heuristic: scout_llm no longer passes the keyword score into the model prompt, so the cheap heuristic and the LLM stay independent signals (their agreement becomes a confidence measure instead of the model just echoing the heuristic). - The judge now scores a rubric per candidate — topical_fit, evidence_match, rigor (0..1 each) and a boolean exclusion_hit — each scored independently rather than as one relative ranking. - New judging.py combines the rubric deterministically in code: a weighted aggregate (weights from topic.json, normalized) with a gentle recency decay -> a 0..10 relevance_score. The model judges criteria; the math does the ranking, so every score is reproducible. - Two-threshold band instead of one cutoff: exclusion_hit vetoes; score >= accept_hi AND heuristic/LLM agreement -> accept; score <= accept_lo -> reject; everything else (mid-score OR signal disagreement) -> uncertain, which stays in the review queue for a human. scout auto-accepts only verdict=accept. - Candidates carry rubric, relevance_score, relevance_confidence, and relevance_verdict; the dashboard candidate cards surface the verdict and confidence. - Offline/heuristic path is unchanged (no rubric -> raw threshold). - Document the judging block in README + topic.example.json; tests for resolve_judging/recency/aggregate and an end-to-end verdict-band run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
Add scripts/eval_metric.py + `make eval` to measure how well the free deterministic relevance metric tracks the LLM judge (LLM score as ground truth) on any labeled candidates.json, so scout runs can lean on the cheap metric and call the LLM far less. Rank-based, scale-free metrics (Spearman, ROC-AUC, precision@K, recall@K, NDCG@K, top-K overlap) across several token-free scorers: the current exact-phrase heuristic, TF-IDF cosine, BM25, IDF-weighted coverage, a naive token-coverage baseline, and a hybrid gate. Plus a routing analysis that reports how many LLM calls a zero-recall-loss prefilter saves. Findings on the bundled 540-paper corpus (docs/metric-vs-llm-eval.md): - The cheap metric's clearly-off-topic bucket held 485/540 candidates and 0 of 8 relevant papers -> prefilter routing cuts ~90% of LLM calls (a hybrid gate ~94%) with full recall. This is the token lever. - No cheap metric reproduces the LLM's fine 7-vs-10 ranking; keep the LLM for the shortlist + uncertainty band only. - Naive token-coverage is worse than random (rewards adjacent-but-wrong neighbours) -> IDF-weight any lexical signal. Pure-Python (no numpy/sklearn). Tests cover the rank statistics, routing recall retention, and full-scorer evaluation. README documents the strategy; report written to a tracked docs/ path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
…eline
Turn the metric-vs-LLM analysis into an actual token saving: before any
LLM call, scout scores candidates with the free deterministic metric and
drops the obvious-off-topic tail, sending only survivors to the judge.
- select_for_llm() ranks by a cheap score and splits into (LLM pool,
prefiltered) with a keep_min floor so a sparse topic never empties.
- Prefiltered candidates stay in candidates.json marked
relevance_verdict="prefiltered" (no silent loss) and never auto-accept.
- Config via judging.prefilter in topic.json {enabled, scorer, threshold,
keep_min}; CLI overrides --prefilter/--no-prefilter,
--prefilter-score, --prefilter-scorer. On by default (current scorer,
threshold 0 = "no positive topical evidence").
- cheap_scores() recomputes the deterministic score from raw fields
(never trusts a stored relevance_score that may hold an LLM verdict),
so the live gate and the eval routing analysis use identical logic.
- eval_metric --recommend/--write-topic auto-derives the highest
zero-recall-loss threshold and writes judging.prefilter back into
topic.json, calibrating the gate to the user's own corpus.
- scout_run records prefiltered_count; run output reports the saving.
On the 540-paper example: 485 dropped, 55 judged (~90% fewer LLM calls)
with all 8 relevant papers retained. Tests cover the gate split, the
disabled path, and a full scout run proving the LLM pool shrinks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MEQUakx19LbZYaqKrcJjbj
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.