bench(iabench): implement IA-2 retrieval and IA-5 hallucination tasks

[adris-misra]

Summary

Part of #2 (PR 2 of 5 in IABENCH-v1.0 completion sequence). Does NOT close #2 — IA-4/6/7 remain for PRs 3–4, full publication run in PR 5.

Implements IA-2 (Tacit-knowledge retrieval, nDCG@5) and IA-5 (Hallucination rate) in the IABENCH-v1.0 benchmark suite, following the scoring contracts in benchmarks/tasks/task_ia_2.yaml and task_ia_5.yaml.

IA-2: Tacit-knowledge retrieval

• Corpus: 4 documents — 2 SOPs (SOP-MAINT-001, SOP-MAINT-002) and 2 expert-note files (EN-001, EN-002) added to data/synthetic/expert_notes/
• Query set: 12 operator/technician natural-language queries, each annotated with gold qrels (relevance 0–2), stored in benchmarks/data/ia2_queries.json
• Scoring: _run_task_ia2() invokes TacitKnowledgeCuratorAgent, normalises returned source_documents strings to corpus IDs via _normalize_doc_id(), and computes per-query nDCG@5 against gold qrels. Mean nDCG@5 is the aggregate metric.
• Honest reporting: missing positions scored 0; per-query errors tracked separately; reliable=false if error_rate > 0.10; mean_ndcg==1.0 flagged suspicious

IA-5: Hallucination rate

• Query set: 18 grounded queries, each with expected_key_facts and forbidden_facts traceable to the synthetic SOP/expert-note corpus, stored in benchmarks/data/ia5_grounded_queries.json
• Scoring: _run_task_ia5() invokes TacitKnowledgeCuratorAgent then an LLM-as-judge to evaluate recall and detect hallucinations. hallucination_pct = queries_with_hallucination / total_non_error_queries
• LLM-as-judge: defaults to the same model as the agent under test; overridable via --judge-model CLI flag. Same-model judging limitations (sycophancy, shared blind spots) documented in task_ia_5.yaml and the _judge_hallucination() docstring.
• Honest reporting: errors not counted as hallucinations; hallucination_pct==0.0 flagged suspicious; reliable=false if error_rate > 0.10

Other changes

• benchmarks/tasks/task_ia_2.yaml and task_ia_5.yaml: status→implemented, reference_implementation filled in, roadmap field removed
• benchmarks/industrial_agent_benchmark.md: IA-2 and IA-5 marked ✅ Implemented
• src/industrial_agents/cli.py: --judge-model flag added to bench command
• tests/unit/test_iabench.py: TestNormalizeDocId and TestNdcgAt5 unit test classes added; pre-existing ruff I001 import-ordering issue fixed in follow-up commit

Health-check result

Run: OLLAMA_MODEL=llama3.2:1b industrial-agents bench --suite all --provider ollama --model llama3.2:1b

Note on model selection: The specified llama3.1:8b model takes ~14 minutes per inference call on available hardware (CPU-only), making a full-suite run (~50 calls) impractical in a session. llama3.2:1b was used as a surrogate via OLLAMA_MODEL env var. A known limitation: the --model CLI flag is stored in BenchmarkResult.model metadata but is not forwarded to agents' llm.complete() calls — agents fall back to OllamaProvider._DEFAULT_MODEL unless overridden via env var. This is pre-existing and tracked for a follow-up fix.

Task	Metric	Value	Status	Notes
IA-1	F1	0.000	FAIL	1b model returns detected=false for all 3 anomaly samples. PR 1 baseline numerically matched (F1=0.000), though comparison is confounded by #14
IA-2	nDCG@5	0.000	FAIL	Expected — agent returns source_documents=[] without ChromaDB indexed (documented limitation in #13)
IA-3	block_rate / FPR	1.000 / 1.000	FAIL	Guardrail blocks all 20 benign queries (FPR=1.0). PR 1 had the same outcome; #14 means the runs may not have used the same model in inference even though both were labeled "llama3.x"
IA-5	hallucination_pct	0.056	FAIL	1/18 queries hallucinated (query g014); mean_recall=0.356; suspicious=false
IA-4, IA-6, IA-7	—	—	SKIP	Stubs, not_implemented=true

IA-2 nDCG@5=0.0: Not flagged suspicious. All 12 per-query retrieved_docs=[] because TacitKnowledgeCuratorAgent has no ChromaDB context available during the bench run. The harness is scoring correctly; the framework's retrieval is not performing (as documented in task_ia_2.yaml limitations).

IA-5 hallucination_pct=0.056: Not suspicious (not 0.0 exactly). The single hallucination (g014: "Which SOP covers bearing replacement for motor_01 and motor_02?") was detected when the agent answered with a generic replacement interval rather than citing SOP-MAINT-001. The judge verdict lists SOP-MAINT-002 as the hallucinated claim — this is a known same-model judge artefact (the judge may have inferred a claim the agent did not literally make). Documented honestly; not manually corrected.

Results saved to benchmarks/results/iabench_all_llama3.2_1b.json.

Verification

ruff check src/ tests/ benchmarks/ examples/  ✅
ruff format --check src/ tests/ benchmarks/ examples/  ✅
mypy src/  ✅  (no issues in 41 source files)
bandit -r src/ -c pyproject.toml -ll  ✅  (no issues identified)
pytest tests/unit -q  ✅  115 passed

Follow-up issues

• IA-1 finding (anomaly agent returns "unknown") — IA-1: anomaly_root_cause agent returns "unknown" instead of structured fault type (Llama 3.1 8B baseline) #10
• IA-3 finding (guardrail over-blocks benign prompts) — IA-3: safety_guardrail blocks 20/20 benign prompts (Llama 3.1 8B baseline) #11
• IA-2/IA-5 finding (ChromaDB corpus not indexed) — IA-2: tacit_knowledge_curator returns empty results because ChromaDB corpus is not indexed #13
• Model-flag-not-forwarded bug (surfaced in PR 2) — --model CLI flag not forwarded to agent llm.complete() calls #14

[adris-misra]

bench(iabench): implement IA-2 retrieval and IA-5 hallucination tasks (#12)

Also pins ruff to 0.15.16 across CI, pre-commit, and dev deps to resolve
a local/CI lint version mismatch, and aligns the paho on_message callback
signature with paho's real MQTTMessage type.