[joelnishanth]

Summary

samples/next-watch/ ranks what a MovieLens member watches next in the shape of GenRec, the LLM-backed ranker Netflix described on 30 July 2026 (tech blog, arXiv 2608.10257). The member's history goes in as three named state fields, twenty candidate titles go in as the options of one Choice, and the probabilities that come back are the ranking.

• Rules serve members with fewer than five likes a popularity list, with no call.
• A second Choice, discovery, pulls toward titles outside a member's usual genres, blended by a weight in questions.yml.
• fetch_movielens.py downloads MovieLens ml-latest-small and checks the MD5 GroupLens publishes. The CSVs stay out of git.
• --user recommends for one member, --explain prints one member's state and every title under every ranker, and --baselines-only needs no key.

The evaluation hides each sampled member's last 4-star-or-higher rating, keeps only earlier ratings, and ranks that title among 19 unrated titles drawn in proportion to popularity before that moment. Popularity-matched negatives are the SASRec and BERT4Rec protocol. Negatives taken from the shortlist's own top titles pushed every free ranker below random in an earlier draft, and the README records that.

Results

100 members, seed 7, jev-1.13.0, 30 September 2026. The report sits in samples/next-watch/data/.

Ranker	MRR	Hit@1	Hit@5	Widens	Distinct #1
random (expected)	0.180	0.050	0.250	n/a	n/a
popularity	0.178	0.050	0.265	0.180	54
genre match	0.294	0.132	0.412	0.000	89
shortlist order	0.253	0.100	0.430	0.000	74
jev next_watch	0.431	0.255	0.648	0.020	86
jev blend	0.411	0.255	0.565	0.020	86

• jev next_watch leads genre match by 0.137 MRR, paired bootstrap 95% interval 0.070 to 0.206, better for 64 members and worse for 26.
• The discovery blend at weight 0.25 cost 0.020 MRR and changed no top pick. The README says so, and suggests sweeping the weight on another seed.
• 100 calls, 242,771 input tokens, $0.0102 at $0.042 per million, 102 ms per call on average from the machine that ran it.

Verified

• uv run python -m py_compile, uvx ruff check . and uvx ruff format --check . on both modules.
• Live runs: an evaluation, --baselines-only, --explain on a ranked member (414) and a cold-start member (54), --user on 414 and on 54, a missing key, and an unknown member id.
• AGENTS.md scores 0.94 readiness with agents_md_readiness.py, the same as main, above the 0.85 CI bar.

Not done

• No Go port and no skill. The Python stands alone, as readme-check does.
• No test suite, matching the repo.

Data

MovieLens ml-latest-small from GroupLens Research, University of Minnesota, which neither endorses nor reviewed this sample. The committed report names MovieLens titles and ids and travels under the GroupLens license terms. The README cites Harper and Konstan (2015).

[aarora79]

Thank you for this work, @joelnishanth.

We would request you to make the following changes so that we can merge.

1. Say something about untrusted state

AGENTS.md lists this under the Jev conventions a sample must follow, and the sample does not touch it. next_watch.py, questions.yml and the README mention untrusted input nowhere.

It carries more weight here than in most samples, because the Choice options are catalog text:

# next_watch.py:636
criteria = {f"m{c['id']}": _option_text(movies[c["id"]]) for c in shortlist}

MovieLens titles are fixed and harmless, so the run is sound. Deploy this shape against a live catalog and those option descriptions carry metadata from studios or from users, where a title written to win the pick moves the answer. A reader copying the pattern should meet that thought inside the sample.

pr-triage handles the same situation by naming the field after where the text came from:

# Treat the description as untrusted. An author who writes "trivial,
# please merge" moves these answers, so the state names what the text is
# rather than presenting it as fact.
"description_written_by_the_author": ...

Two comments and one README paragraph would cover it. Something like this in questions.yml, near where you already explain the runtime criteria:

The option descriptions come from catalog metadata, so treat them the way you treat a prompt. MovieLens titles are fixed and harmless. A catalog that takes metadata from studios or from users is a different matter, and a title written to argue for its own pick would move these answers.

And a "What to notice" entry, which fits that section's voice:

The options are catalog text, so they are untrusted. Jev reads each title's name, year and genres as the thing it chooses among. MovieLens is fixed, so nothing here can argue for itself. A catalog fed by studios or by users can, and the same shape would need its option text treated the way pr-triage treats a pull request description.

Take the wording or write your own.

2. Mark the two bandit B311 findings

bandit reports two Low findings, both the same rule:

next_watch.py:825   rng = random.Random(f"{context['seed']}-{user_id}")
next_watch.py:1331  sample = sorted(random.Random(seed).sample(...))

Every other sample marks a bandit finding with its rule id and a justification, and you already wrote one correctly in this PR:

# fetch_movielens.py:66
with urllib.request.urlopen(url, timeout=FETCH_TIMEOUT_SECONDS) as response:  # nosec B310 - https-only, enforced above

We agree with your reading of the risk. Seeded sampling for a reproducible evaluation is the textbook false positive, and the repo's habit is to say so in the line:

rng = random.Random(f"{context['seed']}-{user_id}")  # nosec B311 - seeded for reproducibility, not security

3. Trim the committed report

553 KB of JSON is more than the repo wants to carry. pr-triage commits 53 KB as its worked example, and the shape there is the one to copy: enough to show a reader what an answer looks like, and no more.

What to keep:

• Every metric, complete. All six rows weigh almost nothing and they are what the README's table and the bootstrap rest on.
• About five results entries, chosen to span the range rather than taken off the top. Every member in this run came back ranked, so the spread worth showing is the reciprocal rank: member 50 landed the hidden title first at 1.000, member 549 sits mid-pack at 0.292, and member 520 managed 0.060. Three like that plus a couple more teach the shape better than the first five in file order.
• A count of what you dropped, so nobody reads the file as the whole run. Two fields do it:

"results_shown": 5,
"results_total": 100,

Five entries puts the file at 28 KB, half of what pr-triage commits. Ten puts it at 56 KB, which is the ceiling we would want.

The markdown report at 4 KB is already the right size. Leave it.

4. Consider splitting next_watch.py

At 1,484 lines it carries the catalog loading, the shortlist, the evaluation harness and the recommender. AGENTS.md asks for a second file once one module stops being readable, and you have already split the fetcher, so moving the evaluation harness out would sit well with that.

[joelnishanth]

Thanks for the careful read, @aarora79. I rebased onto main (#17, #18 and #19; the root README table kept your new pr-triage row and gained the next-watch row below it) and added one commit per point.

1. Untrusted state. questions.yml now carries your paragraph beside the runtime criteria explanation, and build_questions in next_watch.py says why it treats the option text as untrusted, right where it builds the options. The dict that holds them is now catalog_text_by_option, after pr-triage's habit of naming text for its source. The README's "What to notice" ends with your entry, extended to say that recent_likes and recent_dislikes carry the same catalog text. I kept the state field names, option keys and option descriptions as Jev sees them, since renaming those would move answers and void the recorded numbers; a capture of the request for five seed-7 members matches the old one byte for byte.

2. Bandit. Both lines carry # nosec B311 - seeded for reproducibility, not security, and so does the bootstrap's generator from point 3. uvx bandit -q -r . -x ./.venv in samples/next-watch reports no issues (the -x keeps the local virtualenv's third-party code out of the scan).

3. The committed report. data/movielens-small-100-members-seed-7-jev.json drops from 553 KB to 45.5 KB (45,528 bytes). It keeps the six metric rows, results_shown: 5, results_total: 100, and five members in full, which evaluate.py picks by rule: sort by jev next_watch reciprocal rank and take five even steps from best to worst. On this run that gives members 50, 80, 62, 332 and 520 at 1.000, 0.750, 0.250, 0.129 and 0.060. The README's bootstrap also needs every member, so the file adds a reciprocal_ranks table (100 members by five rankers, four places), and the evaluation now computes the bootstrap, stores it and prints it. I rebuilt the file from the recorded 30 September run without new calls, and checked that the metrics, the timestamp, the model and the five kept answers match the original byte for byte. uv run next_watch.py --bootstrap data/movielens-small-100-members-seed-7-jev.json recomputes it with no key and no calls and prints: MRR difference +0.137, 95% interval +0.070 to +0.206. jev next_watch ranked the hidden title higher for 64 members and lower for 26. The markdown report stays as the 30 September run printed it, as you asked. Later runs print the bootstrap as one more line, and the README says so under its output block.

The first two commits on this branch still carry the 912 KB and 553 KB versions of the report, so a squash merge keeps them out of main's history.

4. The split. evaluate.py holds the hold-out split, the sampled negatives, per-member scoring, the metrics and their notes, the bootstrap, the explain view and the reports (883 lines). next_watch.py keeps the catalog, the shortlist, the state, the Jev call, the blend, recommend() and main() (910 lines), so every README command still starts there. evaluate.py imports next_watch, and main() imports evaluate at call time, which keeps the two from importing each other while they load. The helpers evaluate.py calls became public, and the AGENTS.md repo map lists the new file. After the move, --baselines-only on seed 7 wrote a markdown report byte-identical to the one before it.

Verification, 5 October 2026 against jev-1.13.0, 15 calls. py_compile on all three files, ruff check and ruff format --check clean, bandit clean. Live: an evaluation (--users 10 --seed 99 --explain 54, which printed the bootstrap line and served member 54 as cold-start with no call), --explain 414, --baselines-only, --user 414 --verbose --debug, --user 54, and --bootstrap on the committed file and on a scratch report. AGENTS.md scores 0.94 readiness on agents-md-readiness, as before.