examples: replace dynamic-superb fork with audio-hallucination

README.md

@@ -62,7 +62,7 @@ mask wear hung-yi-lee
 | [`hung-yi-lee`](examples/hung-yi-lee) | voice | 李宏毅's Mandarin ML lectures ([@HungyiLeeNTU](https://www.youtube.com/@HungyiLeeNTU)) |
 | [`micrograd`](examples/micrograd) | code | [karpathy/micrograd](https://github.com/karpathy/micrograd) — a tiny autograd engine |
 | [`open-slide`](examples/open-slide) | code | [1weiho/open-slide](https://github.com/1weiho/open-slide) — an agent-native slide framework |
-| [`dynamic-superb`](examples/dynamic-superb) | code | [kuan2jiu99 / Dynamic-SUPERB](https://github.com/kuan2jiu99/dynamic-superb) — a speech benchmark |
+| [`audio-hallucination`](examples/audio-hallucination) | code | [kuan2jiu99/audio-hallucination](https://github.com/kuan2jiu99/audio-hallucination) — eval harnesses for two LALM-hallucination papers |
 
 See the **[gallery](docs/GALLERY.md)** for the exact commands and sample answers.
 

README.zh-TW.md

@@ -60,7 +60,7 @@ mask wear hung-yi-lee
 | [`hung-yi-lee`](examples/hung-yi-lee) | voice | 李宏毅老師的中文 ML 課程（[@HungyiLeeNTU](https://www.youtube.com/@HungyiLeeNTU)）|
 | [`micrograd`](examples/micrograd) | code | [karpathy/micrograd](https://github.com/karpathy/micrograd) —— 極簡 autograd 引擎 |
 | [`open-slide`](examples/open-slide) | code | [1weiho/open-slide](https://github.com/1weiho/open-slide) —— 為 agent 打造的投影片框架 |
-| [`dynamic-superb`](examples/dynamic-superb) | code | [kuan2jiu99 / Dynamic-SUPERB](https://github.com/kuan2jiu99/dynamic-superb) —— 語音 benchmark |
+| [`audio-hallucination`](examples/audio-hallucination) | code | [kuan2jiu99/audio-hallucination](https://github.com/kuan2jiu99/audio-hallucination) —— 兩篇音訊 LLM 幻覺論文的評測工具 |
 
 確切指令與示範回答見 **[gallery](docs/GALLERY.md)**。
 

adapters/agents-md/orchestrator.md

@@ -21,7 +21,7 @@ This agent reads one instruction file (this `AGENTS.md`), so it wears **one mask
 | "update / refresh X, the source changed / has new posts" | `mask redistill X <source>` → extract only the delta digest, merge it into `~/.mask/X/`, then `mask compile X` |
 | "this source is huge / too big to read at once" | opt-in scale mode: `mask scale <workdir>` (headless map-reduce via the user's own agent CLI), then reduce the partials into the mask |
 | "how much does X actually know / its coverage" | `mask coverage X` (also reports knowledge-wiki integrity: orphans, broken `[[links]]`, uncited claims) |
-| "try / show me an example mask / I want to see one" | `mask try <name>` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `dynamic-superb`) into `~/.mask`, then offer to `wear` it |
+| "try / show me an example mask / I want to see one" | `mask try <name>` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `audio-hallucination`) into `~/.mask`, then offer to `wear` it |
 | "wear X" / "answer as X" | `mask wear X`; answer as X thereafter |
 | "ask X: ..." | read X's files and answer as X once, without changing the active mask (see above) |
 | "what masks do I have" | read `~/.mask/_registry.json`, list the roster |

adapters/claude-code/orchestrator.md

@@ -16,7 +16,7 @@ You are now the **mask operator**. Through natural language, the user asks you t
 | "update / refresh X, the source changed / has new posts" | `mask redistill X <source>` → extract only the delta digest, merge it into `~/.mask/X/`, then `mask compile X` |
 | "this source is huge / too big to read at once" | opt-in scale mode: `mask scale <workdir>` (headless map-reduce via the user's own agent CLI), then reduce the partials into the mask |
 | "how much does X actually know / its coverage" | `mask coverage X` (also reports knowledge-wiki integrity: orphans, broken `[[links]]`, uncited claims) |
-| "try / show me an example mask / I want to see one" | `mask try <name>` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `dynamic-superb`) into `~/.mask`, then offer to `wear` it |
+| "try / show me an example mask / I want to see one" | `mask try <name>` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `audio-hallucination`) into `~/.mask`, then offer to `wear` it |
 | "wear X" / "answer as X" | set `~/.mask/_active` to X; answer as X thereafter |
 | "ask X: ..." | read `~/.mask/X/{mask.md, knowledge/, examples.md}` and answer **that one turn** as X with `[src:...]`; do **not** change `_active` (see "Wearing & switching") |
 | "what masks do I have" | read `~/.mask/_registry.json`, list the roster |

docs/GALLERY.md

@@ -86,25 +86,31 @@ sampled, declared out of coverage).
 
 ---
 
-## 🔊 `dynamic-superb` — kuan2jiu99 / Dynamic-SUPERB (code)
+## 🔊 `audio-hallucination` — kuan2jiu99/audio-hallucination (code)
 
-A code-expert mask on the Dynamic-SUPERB speech benchmark (NTU SPML — the same lab
-Hung-yi Lee leads). Instruction-tuned, zero-shot universal speech models.
+A code-expert mask on Chun-Yi Kuan's official code for two NTU papers (with Hung-yi Lee
+— the same lab) on **hallucination in Large Audio-Language Models**: Interspeech 2024
+and ICASSP 2025. The repo ships evaluation harnesses, not a model — you fill in an
+`inference()` stub and score hallucination metrics.
 
 ```sh
-mask ingest https://github.com/kuan2jiu99/dynamic-superb -n dynamic-superb
-mask reduce ~/.mask/.work/dynamic-superb
-mask compile dynamic-superb
+mask ingest https://github.com/kuan2jiu99/audio-hallucination -n audio-hallucination
+mask reduce ~/.mask/.work/audio-hallucination
+# agent extracts the code recipe -> ~/.mask/audio-hallucination/
+mask compile audio-hallucination
 ```
 
-**Evidence:** `repo · 7 files · ~16k chars` · moderate (the evaluation/preprocess API
-+ task layout are well-evidenced; the 33 task definitions and docs were not sampled,
-declared out of coverage).
-
-> **Q: Why is my model scoring 0% when the answers look right?**
-> The accuracy metric is a case-insensitive **exact** match — `pred.lower() == ref.lower()`.
-> "Happy." ≠ "happy" (trailing period), "The emotion is happy" ≠ "happy". Emit exactly
-> the label. `[src:r5]`
+**Evidence:** `repo · 8 files · ~22k chars` · broad (both inference→CSV→evaluation
+harnesses and all metrics — TP/TN/FP/FN, CHAIR/Cover/Hal — are fully in-digest; the
+per-paper sub-READMEs describing the MATCH method, dataset roster, and run commands were
+not sampled, declared out of coverage).
+
+> **Q: My model clearly answers correctly, but accuracy is low — why?**
+> Scoring is a **keyword heuristic**, not semantic. `parse_response()` scans substrings
+> in a fixed order — `"Yes"`→yes, `"No"`/`"does not contain"`→no, `"contain"`→yes,
+> `"not"/"unable"/"can't"`→no — so a hedged "I'm not sure, but it does contain a dog" can
+> trip an earlier branch. In the Interspeech harness an unmatched response even falls
+> through to `""` (there's no final `else`). `[src:r3, r5]`
 
 ---
 

examples/audio-hallucination/examples.md

@@ -0,0 +1,36 @@
+# Examples — audio-hallucination code expert
+
+> Demonstrations of the voice/idiom. Not canonical answers; illustrations.
+
+### Q: How do I plug my audio model into the discriminative harness?
+Fill in the `inference()` stub — that's the entire integration. Everything else (dataset
+loading, CSV writing) is already wired:
+
+```python
+def inference(audio_path, prompt_text):
+    wav = load_audio(audio_path)
+    return my_lalm.generate(wav, prompt_text)   # return raw text; parse_response normalizes it
+```
+
+`inference.py` loops the HF `test` split and writes
+`["entry_id", "audio_index", "label", "response"]` to CSV; then `evaluation.py` scores it.
+You never touch the loop. [src:r4, r8]
+
+### Q: My model clearly answers correctly but accuracy is low — why?
+Scoring is keyword heuristic, not semantic. `parse_response()` looks for substrings in a
+fixed order: `"Yes"/"yes"` → yes, `"No"`/`"does not contain"`/`"there is no"` → no,
+`"contain"` → yes, `"not"/"unable"/"can't"` → no. A verbose answer like *"I cannot be
+certain, but it does contain a dog"* can trip an earlier branch. In the Interspeech harness
+an unmatched response becomes `""` (no `else`), so phrase answers to hit a branch. [src:r3, r5]
+
+### Q: What do CHAIR, Cover, and Hal mean in the generative tasks?
+They are object-set metrics. With `pred` = objects your caption mentions and `label` = the
+ground-truth objects: `CHAIR = 1 − |pred ∩ label| / |pred|` (fraction of mentioned objects
+that are hallucinated), `Cover = |pred ∩ label| / |label|` (fraction of real objects you
+caught), `Hal = 1` if the sample hallucinated at all. Lower CHAIR/Hal, higher Cover is
+better; an empty prediction set yields the `-1` sentinel. [src:r6]
+
+### Q: Why does running the harness unchanged give a wall of "no" predictions?
+Because `inference()` ships as a stub: `pass; return "No"`. Out of the box every prediction
+is the constant `"No"` — a placeholder, not a baseline. Implement the function before
+reading any numbers. [src:r4, r7, r8]

examples/audio-hallucination/knowledge/discriminative-eval.md

@@ -0,0 +1,26 @@
+# Discriminative (yes/no) evaluation
+
+The discriminative harness asks the model a yes/no question about a sound
+("is there a sound of X?") and scores hallucination as a confusion matrix. see also
+[[repo-layout]]. [src:r3, r5]
+
+**Response normalization** — `parse_response(response)` maps free-form model text to a
+label by scanning substrings in a fixed priority order: `"Yes"/"yes"` → `yes`; `"No"` →
+`no`; `"there is no..."` / `"does not contain"` / `"doesn't contain"` → `no`;
+`"contain"/"contains"` → `yes`; `"not"/"unable"/"can't"` → `no`. The **ICASSP** version
+ends with `else: "unknown"`; the **Interspeech** version has no final `else`, so an
+unmatched response is the empty string `""`. [src:r3, r5]
+
+**ICASSP** (`icassp2025/evaluation.py`): `check_answer` turns (response, ground_truth)
+into `TP/TN/FP/FN/unknown`, then: [src:r3]
+- `accuracy = (TP + TN) / total`
+- `precision = TN / (TN + FN)`, `recall = TN / (TN + FP)`, `f1 = 2·p·r/(p+r)`
+- `yes_rate = yes / total`, `instruction_follow_rate = (total − unknown) / total`
+- a **paired before/after analysis**: rows are walked in pairs (`index % 2`) and the
+  transition is bucketed into `TPTN / FNFP / TPFP / FNTN` — this quantifies how a
+  model's answer changes with the stepwise-reasoning (MATCH) prompt. [src:r3]
+
+**Interspeech** (`interspeech2024/evaluation.py`): simpler accuracy, but precision/recall
+are computed on the **negative** class — `precision = answer_right_negative / answer_negative`,
+`recall = answer_right_negative / total_negative` — so they are not the textbook
+positive-class definitions. [src:r5]

examples/audio-hallucination/knowledge/generative-eval.md

@@ -0,0 +1,25 @@
+# Generative (captioning) evaluation
+
+`interspeech2024/generative_tasks/` measures hallucination in free-form audio captions,
+not yes/no answers. see also [[repo-layout]]. [src:r6, r7]
+
+**Inference** (`generative_tasks/inference.py`): each clip is prompted with **5 fixed
+captioning prompts** — `"Describe the audio."`, `"What do you hear?"`, `"What can be
+inferred from the audio?"`, `"This is a sound of"`, `"Generate audio caption:"` — and the
+results are written as a nested JSON `{audio_index: {prompt: {prediction, caption, label,
+task}}}`. [src:r7]
+
+**Metrics** (`generative_tasks/evaluation.py`) operate on Python `set()`s of objects (the
+predicted object set vs the label object set): [src:r6]
+- **CHAIR** (hallucination rate) `= 1 − |pred ∩ label| / |pred|`; returns the sentinel
+  `-1` when the prediction set is empty.
+- **Cover** (coverage) `= |pred ∩ label| / |label|`.
+- **Hal** `= 1 if CHAIR != 0 else 0` (per-sample "did it hallucinate at all"), `-1` when
+  CHAIR is the empty-prediction sentinel.
+
+All three are averaged across every `(yt_id, prompt)` entry and printed as a
+`CHAIR | Cover | Hal` row. Lower CHAIR / Hal and higher Cover is better. [src:r6]
+
+> Note: as sampled, `generative_tasks/inference.py` re-initializes `results = {}` inside
+> the per-clip loop and passes an undefined `audio_path` (it computes `audio_abs_path`) —
+> likely quirks, not asserted intended behavior. [src:r7]

examples/audio-hallucination/knowledge/index.md

@@ -0,0 +1,11 @@
+# Knowledge index
+
+> Topic -> file map. Before stating facts, use this map to find the right file, then grep / read it.
+>
+> This `knowledge/` dir is a small persistent wiki (after Karpathy's LLM-wiki idea): `index.md`
+> is the catalog, `log.md` its chronological companion. Pages carry associative `[[topic]]` links.
+
+- what the repo is (two papers on LALM hallucination) -> overview.md
+- repo layout + the inference→CSV→evaluation conventions -> repo-layout.md
+- the discriminative (yes/no) harness + TP/TN/FP/FN metrics -> discriminative-eval.md   (see also [[repo-layout]])
+- the generative captioning harness + CHAIR/Cover/Hal -> generative-eval.md   (see also [[repo-layout]])

examples/audio-hallucination/knowledge/log.md

@@ -0,0 +1,11 @@
+# Knowledge log
+
+> Append-only, chronological. One entry per distillation / redistillation.
+
+## [2026-06-30] ingest | github.com/kuan2jiu99/audio-hallucination — 8/8 files (r1–r8)
+- ingested README, tree, and the six harness scripts: icassp2025/{evaluation,inference}.py,
+  interspeech2024/{evaluation,inference}.py, interspeech2024/generative_tasks/{evaluation,inference}.py
+- created knowledge/: overview.md, repo-layout.md, discriminative-eval.md, generative-eval.md
+- coverage: broad — the inference→CSV→evaluation harnesses and all metrics are fully
+  evidenced; the two sub-READMEs (MATCH method, dataset roster, run commands) and
+  requirements.txt were NOT sampled, declared out of coverage

examples/audio-hallucination/knowledge/overview.md

@@ -0,0 +1,20 @@
+# What audio-hallucination is
+
+The official code for two NTU papers (Chun-Yi Kuan, Hung-yi Lee, et al.) on
+**hallucination in Large Audio-Language Models (LALMs)** — models that "hear" audio and
+answer in text, but sometimes report sounds/objects that are not actually present. [src:r1]
+
+1. **Interspeech 2024** — "Understanding Sounds, Missing the Questions: The Challenge of
+   Object Hallucination in Large Audio-Language Models" (Kuan, Huang, Lee). The first
+   systematic study of object hallucination in LALMs. arXiv:2406.08402. [src:r1]
+2. **ICASSP 2025** — "Can Large Audio-Language Models Truly Hear? Tackling Hallucinations
+   with Multi-Task Assessment and Stepwise Audio Reasoning" (Kuan, Lee). Covers object
+   existence, temporal order, and object attribute, and proposes **MATCH** (Multi-turn And
+   Thoughtful Chain of Hearings) — a stepwise audio-reasoning prompting method. arXiv:2410.16130. [src:r1]
+
+The repo ships **evaluation harnesses, not a model**: you fill in an `inference()` stub
+with your own audio model, run it over a HuggingFace dataset, and score hallucination
+metrics. See [[repo-layout]] for how the harnesses are structured. [src:r1, r2]
+
+The MATCH method itself, the dataset construction, and exact run commands live in the
+per-paper sub-READMEs, which were not sampled — treat them as out of coverage. [src:r1, r2]

examples/audio-hallucination/knowledge/repo-layout.md

@@ -0,0 +1,34 @@
+# Repo layout & the harness contract
+
+The tree is two self-contained paper directories plus a shared `requirements.txt`: [src:r2]
+
+```
+README.md
+requirements.txt
+icassp2025/
+  evaluation.py          # CSV -> metrics (discriminative + paired analysis)
+  inference.py           # HF dataset -> CSV
+  images/overview.png
+interspeech2024/
+  evaluation.py          # CSV -> metrics (discriminative)
+  inference.py           # HF dataset -> CSV
+  generative_tasks/
+    evaluation.py        # JSON -> CHAIR/Cover/Hal
+    inference.py         # HF dataset -> nested JSON
+  images/...
+```
+
+There is **no shared library** — each script is standalone and argparse-driven, so a
+directory can be copied out on its own. [src:r2]
+
+**The contract** every harness follows: a two-stage split decoupled by a file on disk. [src:r3, r4, r5, r8]
+
+1. `inference.py` defines a stub `def inference(audio_path, prompt_text): pass; return "No"`.
+   You replace its body with your model call. It loops `load_dataset(dataset_name)['test']`,
+   builds `audio_path = f"{audio_root_dir}/{audio_index}"`, and writes results. [src:r4, r8]
+2. `evaluation.py` reads those results back and prints/saves metrics. The two never import
+   each other — the CSV/JSON path is the only interface. [src:r3, r5]
+
+Discriminative results are a 4-column CSV `["entry_id", "audio_index", "label", "response"]`,
+read positionally as `row[0..3]`. Datasets default to the authors' `kuanhuggingface/*` Hub
+repos via `--dataset_name`. see also [[discriminative-eval]] and [[generative-eval]]. [src:r4, r8]