This document drives the next round of fixes for the research pipeline (parse → expand → fetch → candidate-collection → approval → report). Reference run for every failure mode below: 2026-05-07-1908-indian-sme-preferred (query: "indian sme preferred AP/AR software and problems no AP/AR is solving").
Headline numbers from that run: 78 fetched, 78 "approved" (100% pass-through), 8 high-relevance, 27 medium, 43 junk. Precision = 10%. The system fetches well, filters not at all.
Problem: approved.json == candidates.json set-equal in the reference run. There is no LLM call between fetch and approval; llm_calls.jsonl only logs parse and expand. "Approval" is a no-op rename.
Fix: Insert a relevance stage between candidate-collection and approval. For each candidate, call a small/fast model (NOT qwen3:30b — see P1.5) with:
Inputs: original_query, subject, subject_domain, polarity, qualifiers,
candidate.url, candidate.title, first ~500 chars of fetched content
Output: {"verdict": "yes|no|maybe", "reason": "<one line>", "confidence": 0..1}
Drop no. Keep yes. Route maybe through a second pass with full content (truncated to 4k chars).
Log every call to llm_calls.jsonl with stage: "relevance". Persist verdicts to a new relevance.json so we can audit later.
Acceptance:
- Re-running the reference query reduces approved set from 78 → ≤25 with no high-relevance source dropped.
- All 4 BSE filing PDFs, all 4
loginzed.msme.gov.inspam URLs, the APAR Industries page, the AR/VR page, the ARPU LinkedIn post, the SAT/AP-exam page, and the "Subject Matter Expert" LinkedIn Learning page are dropped. - Spec the eight known-good HIGH URLs as a regression set (see Appendix A) — none should be dropped.
Problem: constraints.json.subject_domain is null. Without a domain, the relevance gate has no way to disambiguate "AP/AR" (accounts payable/receivable) from APAR Industries (cables), AR/VR, ARPU, AP exam, and SME ≠ Subject Matter Expert. Five disambiguation failures in one run, all rooted here.
Fix: In the parse prompt, add subject_domain as a required (not nullable) field with a controlled vocabulary: finance_accounting, software_engineering, marketing, hr, supply_chain, legal, healthcare, education, retail, general. The model picks the closest match given subject + topic.
Pass subject_domain to every downstream stage. The relevance gate (P0.1) must reject any candidate whose content is dominantly outside the resolved domain.
Acceptance:
- Reference query resolves
subject_domain: "finance_accounting". - Regression:
"working with SMEs in instructional design"resolvessubject_domain: "education"(not finance). - Add 5 more disambiguation tests as fixtures under
tests/parse/.
Problem: Four candidates in the run have fetch_status: "success" but content is the IIS 404 error template. The fetcher is checking transport, not body.
Fix: Post-fetch validation pass. Mark fetch_status: "invalid_content" (and exclude from candidates) if any of:
- Body contains
IIS 10.0 Detailed Error,HTTP Error 404,Access Denied,Cloudflare,bot-challenged,Just a moment... - Body has < 500 chars of non-boilerplate text (strip nav/footer first)
- Body is > 95% the same as the page title (means "title-only" stub)
web_031.jsonalready hasfetch_error: "success | jina: content invalid or bot-challenged"but isn't excluded — make sure that gets honored.
Acceptance:
- Re-running reference query, all 4 Udyam 404s and the bot-challenged ET page are excluded at fetch stage, not at relevance stage.
Problem: The same Udyam registration page was fetched 4 times: https://www.udyamregistration.gov.in/..., http://udyamregistration.gov.in/... (no www), http://udyamregistration.gov.in/...Msme... (case variant), etc.
Fix: Pre-fetch URL normalization:
- Force
https:// - Strip
www.(or always add — pick one and be consistent) - Lowercase host
- Lowercase path only if the host is known case-insensitive (most are)
- Strip trailing slash, default ports, fragment, common tracking params (
utm_*,fbclid,gclid,ref) - Hash the canonical form; dedupe candidates by hash.
Acceptance: Reference query produces 1 Udyam URL, not 4.
Problem: meta.json for the reference run shows status: "running", completed_at: null — but the run is clearly over (artifacts written, zip downloaded). The frontend can't distinguish "still going" from "crashed silently."
Fix: Wrap the orchestrator in a try/finally. On exception, write status: "failed", capture exception in error, set completed_at. On normal completion, write status: "completed". Add a watchdog: any run with status: "running" and no heartbeat update in > 5 min flips to status: "stalled".
Acceptance: Killing a run mid-flight produces status: "failed" within 5 minutes, not status: "running" forever.
Problem: loginzed.msme.gov.in is an SEO-spam subdomain pushing stock-trend pages. It inherited trust because msme.gov.in is a legitimate government domain. Four candidates from this subdomain made it into approved.
Fix: Maintain domains.yaml with three lists:
allowlist_subdomains: explicit subdomains we trust (www.msme.gov.in,samadhaan.msme.gov.in,udyamregistration.gov.in).blocklist_subdomains: known-bad (loginzed.msme.gov.inand any variant under.gov.inwe don't recognize).allowlist_domains: parent domains where any subdomain is fine (e.g. major newspapers).
Reject any URL on .gov.in whose subdomain isn't on the allowlist. Log to domains_review.jsonl for human review and allowlist promotion.
Acceptance: Reference query produces zero loginzed.msme.gov.in candidates.
Problem: expanded.json shows the LLM produced 6 queries; the file ends with 9. The extra three (site:msme.gov.in, site:udyamregistration.gov.in, site:samadhaan.msme.gov.in) are stapled on unconditionally for any India-tagged query. For "AP/AR software preferences," these gov sites carry zero signal — they're about registration and grievance, not software adoption.
Fix: Replace blanket injection with a topic-conditional table:
india_sme_topic_sites:
policy_scheme: [msme.gov.in, ramp.msme.gov.in]
registration: [udyamregistration.gov.in]
grievance_payment: [samadhaan.msme.gov.in]
software_tooling: [] # no gov sites — search the open web
finance_accounting: []
hiring: [naukri.com, linkedin.com]The expander selects sets based on topic + subject_domain from the parser. Empty list = no injection.
Acceptance: Reference query (topic="unsolved problems", subject_domain="finance_accounting") produces 0 hardcoded gov-site queries.
Problem: The reference query asks two things: (a) preferred software (positive), (b) problems no AP/AR is solving (negative). Parser collapsed to single intent=SENTIMENT, polarity=negative. All 9 expanded queries are negative-framed ("problems," "challenges," "issues," "pain points"). The "preferred" half got zero coverage.
Fix: Add a decompose step before parse. Prompt:
"Does this query contain multiple distinct sub-questions (different intents, different polarities, or
X and Ystructure with X and Y being separately-answerable)? If yes, return a list of self-contained sub-queries. If no, return the original."
Run the full pipeline per sub-query. Merge approved sets at the end with provenance tags so the report can answer each half.
Acceptance:
- Reference query decomposes into 2 sub-queries (preference + pain).
- Final report has a section for "preferred software" and a section for "unsolved problems."
- Regression: single-polarity queries (e.g. "what's wrong with Tally") don't decompose.
Problem: Reference run pulled 7 BSE filing PDFs (~10 MB total) including a 374-page NTPC annual report. URL pattern bsmedia.business-standard.com/_media/bs/data/announcements/bse/.*\.pdf is reliably noise for our use cases — these are stock-exchange disclosure documents, not industry research.
Fix: url_blocklist_patterns.yaml with regex patterns for known-noise paths:
- pattern: "bsmedia\\.business-standard\\.com/_media/bs/data/announcements/bse/.*\\.pdf"
reason: "BSE corporate filings, no industry signal"
- pattern: "linkedin\\.com/jobs/view/"
reason: "Job listings, not problem discussions"
- pattern: "linkedin\\.com/learning/"
reason: "Course pages, marketing content"
- pattern: "economictimes\\.indiatimes\\.com/.*/stocksupdate/"
reason: "Stock price updates"Reject pre-fetch. Saves bandwidth, storage, and downstream LLM cost.
Acceptance: Reference query fetches 0 BSE filing PDFs, 0 LinkedIn job posts, 0 LinkedIn Learning pages.
Problem: parse took 54s, expand took 119s. Both produce <150 tokens of structured JSON. We're using a 30B model to do JSON shaping a 3B model handles in 3s.
Fix: Move parse, expand, decompose, relevance to a small fast model (qwen3:7b, llama3.1:8b, or comparable). Keep the big model only for the final report synthesis where reasoning depth actually matters. Add per-stage model config in pipeline.yaml:
stages:
decompose: { model: qwen3:7b, timeout_s: 30 }
parse: { model: qwen3:7b, timeout_s: 30 }
expand: { model: qwen3:7b, timeout_s: 30 }
relevance: { model: qwen3:7b, timeout_s: 15 }
report: { model: qwen3:30b-a3b, timeout_s: 300 }Acceptance: Total parse + expand latency drops from ~3 min to <15s. Output quality on a 20-query fixture suite is within 5% of the 30B baseline (measure: % of fixtures producing same subject_domain and >=80% query overlap).
Problem: In the reference run, Reddit's hit rate (genuine signal vs total fetched) was visibly higher than the open web's. Eyeballing: ~5/10 Reddit URLs are HIGH, vs 3/68 web URLs HIGH. Yet the pipeline treats them as a single pool.
Fix: Tag each candidate with source_class (reddit, linkedin_pulse, news_indian, news_global, vendor_blog, gov, academic, forum_other). Weight retrieval and budget by class:
- For "Indian SME pain" queries, allocate ~50% of fetch budget to Reddit + Indian forums + LinkedIn Pulse from Indian authors.
- For "preferred software" queries, allocate more to vendor blogs + product-listing sites (G2, Capterra) + LinkedIn product pages.
Make these weights live in pipeline.yaml keyed by (subject_domain, intent, geographic_qualifier).
Acceptance: Reference query allocates ≥40% of fetch budget to Reddit + Indian forums.
Even after the gate (P0.1), keep the score (0–1) on each approved candidate. The report stage should weight quotes by source relevance.
meta.json should include tokens_in_total, tokens_out_total, latency_ms_total, bytes_fetched_total. Surface in the UI. We need to know per-run unit economics before we charge anyone.
If relevance fails halfway through, we shouldn't re-fetch 78 URLs to resume. Persist intermediate state per stage; on retry, skip stages whose output already exists and is valid.
Use the Reddit JSON API directly (/.json suffix) with retries. The current generic-fetcher path works but loses comments/scores in some cases (note metadata has truncated flag — verify it's honored). Comments are where the actual product preference signals live.
Hard cap per-fetch body size at 1 MB by default; for application/pdf, 500 KB. URL patterns that look like academic dissertations or annual reports either (a) pre-rejected (P1.4) or (b) summarized via a single LLM extraction call rather than stored verbatim.
Use this set as a relevance-gate fixture. The eight HIGH-relevance URLs should pass the gate; the listed LOW URLs should be rejected. Keep these in tests/fixtures/regression_indian_sme_apar.yaml.
Must-pass (HIGH):
reddit.com/r/StartUpIndia/comments/1pt6354/why_indian_smes_arent_adopting_ai_and_my_plan_toreddit.com/r/StartUpIndia/comments/1doxoxx/why_has_no_one_distrupted_accounting_market_yetreddit.com/r/IndiaBusiness/comments/1mul6di/curious_to_know_what_exactly_stops_small_indianreddit.com/r/CharteredAccountants/comments/1ie6sh5/accounting_software_relevant_for_indiareddit.com/r/developersIndia/comments/1irm7r4/what_problems_do_businesses_face_with_accountingreddit.com/r/indianstartups/comments/1oserhn/these_gst_rules_are_slowly_killing_my_smallopen.money/blog/invoice-exception-handling-best-practicesaiaccountant.com/blog/automated-statement-reconciliation-india
Must-reject (LOW, sample):
economictimes.indiatimes.com/topic/apar-technologies(APAR Industries — cables, not AP/AR)economictimes.indiatimes.com/small-biz/.../indian-firms-plan-to-leverage-on-ar-vr-experiences/...(AR/VR)linkedin.com/pulse/youtube-make-originals-india-boost-arpu-...(ARPU, not AR)economictimes.indiatimes.com/nri/test-guide/demystifying-the-sat-and-ap-programs-...(AP exam)linkedin.com/learning/instructional-design-working-with-smes/what-is-a-sme(Subject Matter Expert)- Any of the 4 Udyam registration 404s
- Any
loginzed.msme.gov.in/* - Any
bsmedia.business-standard.com/_media/bs/data/announcements/bse/*.pdf
A new run on the reference query (indian sme preferred AP/AR software and problems no AP/AR is solving) must satisfy:
meta.json.status == "completed"withcompleted_atpopulated.approved.urlscontains all 8 HIGH-regression URLs.approved.urlscontains zero of the listed must-reject URLs.approved.urlslength ≤ 25 (precision floor: 8/25 = 32%, vs current 8/78 = 10%).- Total run wall-clock time ≤ 4 min (vs current ~5+ min for an incomplete run).
llm_calls.jsonlhas at least 4 stages logged:decompose,parse,expand,relevance, plus per-candidate relevance entries.- Final report has separate sections answering "preferred software" and "unsolved problems."