Task 5: demand-aware ingestion funnel

[mircealungu]

Makes the crawler's expensive LLM simplification demand-driven instead of supply-driven, and gives newly-demanded buckets an immediate head start. All 6 phases of the Task 5 design.

Problem

Simplification ran until a static per-language cap (50/day/topic for da/fr/de, 20 otherwise), then stopped. So dormant languages were simplified at the same rate as busy ones, and whichever articles appeared first in feed order won the quota (first-N, not best-N). Evidence from the week-28 report: ~370 crawled per 1 opened.

What changed

New zeeguu/core/content_retriever/funnel.py

• compute_demand_surface() — the distinct (language, topic) buckets that active readers (last_seen ≤ 30d) actually want, deduped across the cohort. A reader with no topic subscription reads the whole language, so they lend demand to every topic in it.
• FunnelBudget — per-(language, topic) quota that scales 20→50 with reader count, plus a pilot-light floor (≤5 distinct topics/day) for dormant supported languages so a brand-new learner (or the Romanian kiosk) never opens an empty feed.
• Pre-download title triage — before paying any readability/download cost: skip a feed entirely when its language is already satisfied for the day, or keep only the best-N titles when headroom is limited. Cheap LLM ranker with a safe first-N fallback (only runs when candidates exceed the keep count), so triage can only reduce waste, never drop a must-have article.

article_downloader.py + crawl.py thread a FunnelBudget instead of the old static-cap counts dict. Legacy download_from_feed callers pass no budget → behaviour unchanged. No-topic articles are now simplified only when the language has all-topic readers (old code always simplified them = pure waste).

Promotion-on-join — backfill.py
Subscribing to a topic or switching learned language fires a background backfill that simplifies a few existing recent raw articles for the newly-demanded bucket. Dormant buckets already hold raw, un-simplified articles (the crawl stores them, only skips the LLM), so this is fast and download-free — no mini-crawl. No-ops if the bucket already has fresh inventory. Wired into the subscribe and learned-language endpoints via run_in_background.

Phase mapping

Phase	Status
1. Demand surface + quota	done
2. Two-tier pilot-light floor	done
3. Title-triage best-N	done (pre-download LLM triage)
4. Pre-simplify cache	already satisfied (crawl-time simplification = the cache; /simplify_article = the tail)
5. Promotion-on-join	done
6. Romanian floor	satisfied by the floor covering every supported language

Tests

31 tests in zeeguu/core/test/test_funnel.py (in-memory SQLite): quota scaling, demand surface, gate/floor/no-topic paths, headroom, triage selection + fallback, backfill inventory queries. Regression-checked test_retrieve_and_compute / test_article / test_feed / test_endpoint_names.

Reviewer notes / conscious choices

• Active reader = last_seen ≤ 30d (chosen over open-based). Implication: German (many recently-seen users) stays "in demand" — this does not reproduce the open-based "German → 0" reduction; topic-dedup and dormant-language savings still apply. The definition is isolated in one function (_active_reader_rows) for an easy switch later.
• Not yet validated on a real crawl / real subscribe flow — logic is unit-tested with the LLM mocked; the LLM ranker and background backfill need a live run.
• crawl_stats.py dashboard still displays the now-superseded static "cap" (legacy constants kept alive only for it) — follow-up.
• Backfill has no dedup guard against concurrent triggers (conservative caps mitigate); a brand-new never-crawled language finds no raw inventory and relies on the floor + next crawl.

🤖 Generated with Claude Code

[github-actions]

ArchLens detected architectural changes in the following views:

[mircealungu]

Closing — superseded rather than rejected. Two things changed under it since 15 Jul.

1. The pipeline moved. #698 made simplification on-demand, so the expensive crawl-time step is no longer simplify_and_classify but assess_and_summarize. The should_simplify gate in download_feed_item now sits in front of assessment, so a gated article gets no cefr_level, no summary and no per-level text — and it still reaches the feed, where it renders as the publisher's blurb at every CEFR level. That is the exact bug reported this week. It would also park the new assessment-coverage alarm (#718) around 73% against an 80% threshold, mailing daily about intended behaviour.

2. The budget has a cliff. language_simplification_headroom returns None — unbounded — as soon as one active reader of a language has no topic subscription. Measured today:

language	active readers	unfiltered	triage
de	5	0	prunes
it / el / ro / pt	6 / 4 / 3 / 1	0	prunes
fr	15	4	no pruning
en / da / es / nl	11 / 11 / 9 / 7	1 / 1 / 2 / 1	no pruning

So it would bite on 55% of volume (including German, the single biggest line item) and do nothing for the other 45% — and one user clearing their topic subscriptions silently flips a language back to unbounded, with no signal.

What replaces it. A fill-to-target model: for each of the 72 currently demanded (language, topic) buckets, ensure ~3 fresh articles that the bucket's active readers haven't seen; assess only the deficit. That is ~216/day against ~1,324 today (~84%), it is self-limiting (a quiet day costs nothing, where a quota always spends its allowance), and "reader with no topics" becomes all 8 topics need filling rather than infinity — the cliff disappears instead of being patched.

Worth lifting out of this PR when that lands:

• compute_demand_surface() — the demand-surface query is needed unchanged.
• backfill.py promotion-on-join — still the right behaviour when someone subscribes or switches language.
• The title ranker in select_titles_to_download. Under a target model which articles fill the deficit matters much more than under a quota, so this becomes more useful, not less — repointed at interestingness/diversity rather than at best-N-for-quota.

Thanks — the demand-surface framing is what made the replacement obvious.