From 07a42cfd72bfaf3b8efab47d9ef5194381adec1d Mon Sep 17 00:00:00 2001 From: kuanpo Date: Tue, 30 Jun 2026 14:02:33 +0800 Subject: [PATCH] examples: replace dynamic-superb fork with audio-hallucination MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dynamic-superb was a fork; swap it for kuan2jiu99/audio-hallucination, an original repo (eval harnesses for two LALM-hallucination papers). - examples/audio-hallucination/: new code mask distilled via the code recipe — mask.md (6-section conventions profile), a 4-page knowledge wiki (overview/repo-layout/discriminative-eval/generative-eval) with index + log, examples.md, and sources.json (8 sources, real hashes). Every claim is [src:id]-bound; passes `mask coverage` broad with zero orphans, broken links, or unknown citations. - examples/dynamic-superb/: removed (fork, not original). - README.md / README.zh-TW.md / docs/GALLERY.md: gallery rows + section rewritten for the new mask. - adapters/{claude-code,agents-md}/orchestrator.md: `mask try` example roster updated. - site/index.html: sync landing page to v0.3.0 — add a "See it in action" four-mask matrix, align Quickstart with the install.sh/mask flow, and surface the knowledge-wiki + coverage-integrity features. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- README.zh-TW.md | 2 +- adapters/agents-md/orchestrator.md | 2 +- adapters/claude-code/orchestrator.md | 2 +- docs/GALLERY.md | 34 +++--- examples/audio-hallucination/examples.md | 36 ++++++ .../knowledge/discriminative-eval.md | 26 +++++ .../knowledge/generative-eval.md | 25 +++++ .../audio-hallucination/knowledge/index.md | 11 ++ examples/audio-hallucination/knowledge/log.md | 11 ++ .../audio-hallucination/knowledge/overview.md | 20 ++++ .../knowledge/repo-layout.md | 34 ++++++ examples/audio-hallucination/mask.md | 106 ++++++++++++++++++ examples/audio-hallucination/sources.json | 106 ++++++++++++++++++ examples/dynamic-superb/examples.md | 33 ------ .../dynamic-superb/knowledge/evaluation.md | 25 ----- examples/dynamic-superb/knowledge/index.md | 11 -- examples/dynamic-superb/knowledge/log.md | 11 -- examples/dynamic-superb/knowledge/overview.md | 15 --- .../dynamic-superb/knowledge/preprocess.md | 19 ---- .../dynamic-superb/knowledge/task-format.md | 23 ---- examples/dynamic-superb/mask.md | 91 --------------- examples/dynamic-superb/sources.json | 94 ---------------- site/index.html | 25 ++++- 24 files changed, 420 insertions(+), 344 deletions(-) create mode 100644 examples/audio-hallucination/examples.md create mode 100644 examples/audio-hallucination/knowledge/discriminative-eval.md create mode 100644 examples/audio-hallucination/knowledge/generative-eval.md create mode 100644 examples/audio-hallucination/knowledge/index.md create mode 100644 examples/audio-hallucination/knowledge/log.md create mode 100644 examples/audio-hallucination/knowledge/overview.md create mode 100644 examples/audio-hallucination/knowledge/repo-layout.md create mode 100644 examples/audio-hallucination/mask.md create mode 100644 examples/audio-hallucination/sources.json delete mode 100644 examples/dynamic-superb/examples.md delete mode 100644 examples/dynamic-superb/knowledge/evaluation.md delete mode 100644 examples/dynamic-superb/knowledge/index.md delete mode 100644 examples/dynamic-superb/knowledge/log.md delete mode 100644 examples/dynamic-superb/knowledge/overview.md delete mode 100644 examples/dynamic-superb/knowledge/preprocess.md delete mode 100644 examples/dynamic-superb/knowledge/task-format.md delete mode 100644 examples/dynamic-superb/mask.md delete mode 100644 examples/dynamic-superb/sources.json diff --git a/README.md b/README.md index 18f1fe4..5f033b5 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ mask wear hung-yi-lee | [`hung-yi-lee`](examples/hung-yi-lee) | voice | 李宏毅's Mandarin ML lectures ([@HungyiLeeNTU](https://www.youtube.com/@HungyiLeeNTU)) | | [`micrograd`](examples/micrograd) | code | [karpathy/micrograd](https://github.com/karpathy/micrograd) — a tiny autograd engine | | [`open-slide`](examples/open-slide) | code | [1weiho/open-slide](https://github.com/1weiho/open-slide) — an agent-native slide framework | -| [`dynamic-superb`](examples/dynamic-superb) | code | [kuan2jiu99 / Dynamic-SUPERB](https://github.com/kuan2jiu99/dynamic-superb) — a speech benchmark | +| [`audio-hallucination`](examples/audio-hallucination) | code | [kuan2jiu99/audio-hallucination](https://github.com/kuan2jiu99/audio-hallucination) — eval harnesses for two LALM-hallucination papers | See the **[gallery](docs/GALLERY.md)** for the exact commands and sample answers. diff --git a/README.zh-TW.md b/README.zh-TW.md index 539cf7b..116442c 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -60,7 +60,7 @@ mask wear hung-yi-lee | [`hung-yi-lee`](examples/hung-yi-lee) | voice | 李宏毅老師的中文 ML 課程（[@HungyiLeeNTU](https://www.youtube.com/@HungyiLeeNTU)）| | [`micrograd`](examples/micrograd) | code | [karpathy/micrograd](https://github.com/karpathy/micrograd) —— 極簡 autograd 引擎 | | [`open-slide`](examples/open-slide) | code | [1weiho/open-slide](https://github.com/1weiho/open-slide) —— 為 agent 打造的投影片框架 | -| [`dynamic-superb`](examples/dynamic-superb) | code | [kuan2jiu99 / Dynamic-SUPERB](https://github.com/kuan2jiu99/dynamic-superb) —— 語音 benchmark | +| [`audio-hallucination`](examples/audio-hallucination) | code | [kuan2jiu99/audio-hallucination](https://github.com/kuan2jiu99/audio-hallucination) —— 兩篇音訊 LLM 幻覺論文的評測工具 | 確切指令與示範回答見 **[gallery](docs/GALLERY.md)**。 diff --git a/adapters/agents-md/orchestrator.md b/adapters/agents-md/orchestrator.md index d2fd0b3..5476290 100644 --- a/adapters/agents-md/orchestrator.md +++ b/adapters/agents-md/orchestrator.md @@ -21,7 +21,7 @@ This agent reads one instruction file (this `AGENTS.md`), so it wears **one mask | "update / refresh X, the source changed / has new posts" | `mask redistill X ` → extract only the delta digest, merge it into `~/.mask/X/`, then `mask compile X` | | "this source is huge / too big to read at once" | opt-in scale mode: `mask scale ` (headless map-reduce via the user's own agent CLI), then reduce the partials into the mask | | "how much does X actually know / its coverage" | `mask coverage X` (also reports knowledge-wiki integrity: orphans, broken `[[links]]`, uncited claims) | -| "try / show me an example mask / I want to see one" | `mask try ` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `dynamic-superb`) into `~/.mask`, then offer to `wear` it | +| "try / show me an example mask / I want to see one" | `mask try ` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `audio-hallucination`) into `~/.mask`, then offer to `wear` it | | "wear X" / "answer as X" | `mask wear X`; answer as X thereafter | | "ask X: ..." | read X's files and answer as X once, without changing the active mask (see above) | | "what masks do I have" | read `~/.mask/_registry.json`, list the roster | diff --git a/adapters/claude-code/orchestrator.md b/adapters/claude-code/orchestrator.md index 313116d..8c010cc 100644 --- a/adapters/claude-code/orchestrator.md +++ b/adapters/claude-code/orchestrator.md @@ -16,7 +16,7 @@ You are now the **mask operator**. Through natural language, the user asks you t | "update / refresh X, the source changed / has new posts" | `mask redistill X ` → extract only the delta digest, merge it into `~/.mask/X/`, then `mask compile X` | | "this source is huge / too big to read at once" | opt-in scale mode: `mask scale ` (headless map-reduce via the user's own agent CLI), then reduce the partials into the mask | | "how much does X actually know / its coverage" | `mask coverage X` (also reports knowledge-wiki integrity: orphans, broken `[[links]]`, uncited claims) | -| "try / show me an example mask / I want to see one" | `mask try ` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `dynamic-superb`) into `~/.mask`, then offer to `wear` it | +| "try / show me an example mask / I want to see one" | `mask try ` — install a curated example (`hung-yi-lee`, `micrograd`, `open-slide`, `audio-hallucination`) into `~/.mask`, then offer to `wear` it | | "wear X" / "answer as X" | set `~/.mask/_active` to X; answer as X thereafter | | "ask X: ..." | read `~/.mask/X/{mask.md, knowledge/, examples.md}` and answer **that one turn** as X with `[src:...]`; do **not** change `_active` (see "Wearing & switching") | | "what masks do I have" | read `~/.mask/_registry.json`, list the roster | diff --git a/docs/GALLERY.md b/docs/GALLERY.md index e67cd6e..049b342 100644 --- a/docs/GALLERY.md +++ b/docs/GALLERY.md @@ -86,25 +86,31 @@ sampled, declared out of coverage). --- -## 🔊 `dynamic-superb` — kuan2jiu99 / Dynamic-SUPERB (code) +## 🔊 `audio-hallucination` — kuan2jiu99/audio-hallucination (code) -A code-expert mask on the Dynamic-SUPERB speech benchmark (NTU SPML — the same lab -Hung-yi Lee leads). Instruction-tuned, zero-shot universal speech models. +A code-expert mask on Chun-Yi Kuan's official code for two NTU papers (with Hung-yi Lee +— the same lab) on **hallucination in Large Audio-Language Models**: Interspeech 2024 +and ICASSP 2025. The repo ships evaluation harnesses, not a model — you fill in an +`inference()` stub and score hallucination metrics. ```sh -mask ingest https://github.com/kuan2jiu99/dynamic-superb -n dynamic-superb -mask reduce ~/.mask/.work/dynamic-superb -mask compile dynamic-superb +mask ingest https://github.com/kuan2jiu99/audio-hallucination -n audio-hallucination +mask reduce ~/.mask/.work/audio-hallucination +# agent extracts the code recipe -> ~/.mask/audio-hallucination/ +mask compile audio-hallucination ``` -**Evidence:** `repo · 7 files · ~16k chars` · moderate (the evaluation/preprocess API -+ task layout are well-evidenced; the 33 task definitions and docs were not sampled, -declared out of coverage). - -> **Q: Why is my model scoring 0% when the answers look right?** -> The accuracy metric is a case-insensitive **exact** match — `pred.lower() == ref.lower()`. -> "Happy." ≠ "happy" (trailing period), "The emotion is happy" ≠ "happy". Emit exactly -> the label. `[src:r5]` +**Evidence:** `repo · 8 files · ~22k chars` · broad (both inference→CSV→evaluation +harnesses and all metrics — TP/TN/FP/FN, CHAIR/Cover/Hal — are fully in-digest; the +per-paper sub-READMEs describing the MATCH method, dataset roster, and run commands were +not sampled, declared out of coverage). + +> **Q: My model clearly answers correctly, but accuracy is low — why?** +> Scoring is a **keyword heuristic**, not semantic. `parse_response()` scans substrings +> in a fixed order — `"Yes"`→yes, `"No"`/`"does not contain"`→no, `"contain"`→yes, +> `"not"/"unable"/"can't"`→no — so a hedged "I'm not sure, but it does contain a dog" can +> trip an earlier branch. In the Interspeech harness an unmatched response even falls +> through to `""` (there's no final `else`). `[src:r3, r5]` --- diff --git a/examples/audio-hallucination/examples.md b/examples/audio-hallucination/examples.md new file mode 100644 index 0000000..7c86aa4 --- /dev/null +++ b/examples/audio-hallucination/examples.md @@ -0,0 +1,36 @@ +# Examples — audio-hallucination code expert + +> Demonstrations of the voice/idiom. Not canonical answers; illustrations. + +### Q: How do I plug my audio model into the discriminative harness? +Fill in the `inference()` stub — that's the entire integration. Everything else (dataset +loading, CSV writing) is already wired: + +```python +def inference(audio_path, prompt_text): + wav = load_audio(audio_path) + return my_lalm.generate(wav, prompt_text) # return raw text; parse_response normalizes it +``` + +`inference.py` loops the HF `test` split and writes +`["entry_id", "audio_index", "label", "response"]` to CSV; then `evaluation.py` scores it. +You never touch the loop. [src:r4, r8] + +### Q: My model clearly answers correctly but accuracy is low — why? +Scoring is keyword heuristic, not semantic. `parse_response()` looks for substrings in a +fixed order: `"Yes"/"yes"` → yes, `"No"`/`"does not contain"`/`"there is no"` → no, +`"contain"` → yes, `"not"/"unable"/"can't"` → no. A verbose answer like *"I cannot be +certain, but it does contain a dog"* can trip an earlier branch. In the Interspeech harness +an unmatched response becomes `""` (no `else`), so phrase answers to hit a branch. [src:r3, r5] + +### Q: What do CHAIR, Cover, and Hal mean in the generative tasks? +They are object-set metrics. With `pred` = objects your caption mentions and `label` = the +ground-truth objects: `CHAIR = 1 − |pred ∩ label| / |pred|` (fraction of mentioned objects +that are hallucinated), `Cover = |pred ∩ label| / |label|` (fraction of real objects you +caught), `Hal = 1` if the sample hallucinated at all. Lower CHAIR/Hal, higher Cover is +better; an empty prediction set yields the `-1` sentinel. [src:r6] + +### Q: Why does running the harness unchanged give a wall of "no" predictions? +Because `inference()` ships as a stub: `pass; return "No"`. Out of the box every prediction +is the constant `"No"` — a placeholder, not a baseline. Implement the function before +reading any numbers. [src:r4, r7, r8] diff --git a/examples/audio-hallucination/knowledge/discriminative-eval.md b/examples/audio-hallucination/knowledge/discriminative-eval.md new file mode 100644 index 0000000..5941c98 --- /dev/null +++ b/examples/audio-hallucination/knowledge/discriminative-eval.md @@ -0,0 +1,26 @@ +# Discriminative (yes/no) evaluation + +The discriminative harness asks the model a yes/no question about a sound +("is there a sound of X?") and scores hallucination as a confusion matrix. see also +[[repo-layout]]. [src:r3, r5] + +**Response normalization** — `parse_response(response)` maps free-form model text to a +label by scanning substrings in a fixed priority order: `"Yes"/"yes"` → `yes`; `"No"` → +`no`; `"there is no..."` / `"does not contain"` / `"doesn't contain"` → `no`; +`"contain"/"contains"` → `yes`; `"not"/"unable"/"can't"` → `no`. The **ICASSP** version +ends with `else: "unknown"`; the **Interspeech** version has no final `else`, so an +unmatched response is the empty string `""`. [src:r3, r5] + +**ICASSP** (`icassp2025/evaluation.py`): `check_answer` turns (response, ground_truth) +into `TP/TN/FP/FN/unknown`, then: [src:r3] +- `accuracy = (TP + TN) / total` +- `precision = TN / (TN + FN)`, `recall = TN / (TN + FP)`, `f1 = 2·p·r/(p+r)` +- `yes_rate = yes / total`, `instruction_follow_rate = (total − unknown) / total` +- a **paired before/after analysis**: rows are walked in pairs (`index % 2`) and the + transition is bucketed into `TPTN / FNFP / TPFP / FNTN` — this quantifies how a + model's answer changes with the stepwise-reasoning (MATCH) prompt. [src:r3] + +**Interspeech** (`interspeech2024/evaluation.py`): simpler accuracy, but precision/recall +are computed on the **negative** class — `precision = answer_right_negative / answer_negative`, +`recall = answer_right_negative / total_negative` — so they are not the textbook +positive-class definitions. [src:r5] diff --git a/examples/audio-hallucination/knowledge/generative-eval.md b/examples/audio-hallucination/knowledge/generative-eval.md new file mode 100644 index 0000000..40dbf48 --- /dev/null +++ b/examples/audio-hallucination/knowledge/generative-eval.md @@ -0,0 +1,25 @@ +# Generative (captioning) evaluation + +`interspeech2024/generative_tasks/` measures hallucination in free-form audio captions, +not yes/no answers. see also [[repo-layout]]. [src:r6, r7] + +**Inference** (`generative_tasks/inference.py`): each clip is prompted with **5 fixed +captioning prompts** — `"Describe the audio."`, `"What do you hear?"`, `"What can be +inferred from the audio?"`, `"This is a sound of"`, `"Generate audio caption:"` — and the +results are written as a nested JSON `{audio_index: {prompt: {prediction, caption, label, +task}}}`. [src:r7] + +**Metrics** (`generative_tasks/evaluation.py`) operate on Python `set()`s of objects (the +predicted object set vs the label object set): [src:r6] +- **CHAIR** (hallucination rate) `= 1 − |pred ∩ label| / |pred|`; returns the sentinel + `-1` when the prediction set is empty. +- **Cover** (coverage) `= |pred ∩ label| / |label|`. +- **Hal** `= 1 if CHAIR != 0 else 0` (per-sample "did it hallucinate at all"), `-1` when + CHAIR is the empty-prediction sentinel. + +All three are averaged across every `(yt_id, prompt)` entry and printed as a +`CHAIR | Cover | Hal` row. Lower CHAIR / Hal and higher Cover is better. [src:r6] + +> Note: as sampled, `generative_tasks/inference.py` re-initializes `results = {}` inside +> the per-clip loop and passes an undefined `audio_path` (it computes `audio_abs_path`) — +> likely quirks, not asserted intended behavior. [src:r7] diff --git a/examples/audio-hallucination/knowledge/index.md b/examples/audio-hallucination/knowledge/index.md new file mode 100644 index 0000000..03a8134 --- /dev/null +++ b/examples/audio-hallucination/knowledge/index.md @@ -0,0 +1,11 @@ +# Knowledge index + +> Topic -> file map. Before stating facts, use this map to find the right file, then grep / read it. +> +> This `knowledge/` dir is a small persistent wiki (after Karpathy's LLM-wiki idea): `index.md` +> is the catalog, `log.md` its chronological companion. Pages carry associative `[[topic]]` links. + +- what the repo is (two papers on LALM hallucination) -> overview.md +- repo layout + the inference→CSV→evaluation conventions -> repo-layout.md +- the discriminative (yes/no) harness + TP/TN/FP/FN metrics -> discriminative-eval.md (see also [[repo-layout]]) +- the generative captioning harness + CHAIR/Cover/Hal -> generative-eval.md (see also [[repo-layout]]) diff --git a/examples/audio-hallucination/knowledge/log.md b/examples/audio-hallucination/knowledge/log.md new file mode 100644 index 0000000..7445f66 --- /dev/null +++ b/examples/audio-hallucination/knowledge/log.md @@ -0,0 +1,11 @@ +# Knowledge log + +> Append-only, chronological. One entry per distillation / redistillation. + +## [2026-06-30] ingest | github.com/kuan2jiu99/audio-hallucination — 8/8 files (r1–r8) +- ingested README, tree, and the six harness scripts: icassp2025/{evaluation,inference}.py, + interspeech2024/{evaluation,inference}.py, interspeech2024/generative_tasks/{evaluation,inference}.py +- created knowledge/: overview.md, repo-layout.md, discriminative-eval.md, generative-eval.md +- coverage: broad — the inference→CSV→evaluation harnesses and all metrics are fully + evidenced; the two sub-READMEs (MATCH method, dataset roster, run commands) and + requirements.txt were NOT sampled, declared out of coverage diff --git a/examples/audio-hallucination/knowledge/overview.md b/examples/audio-hallucination/knowledge/overview.md new file mode 100644 index 0000000..c65527e --- /dev/null +++ b/examples/audio-hallucination/knowledge/overview.md @@ -0,0 +1,20 @@ +# What audio-hallucination is + +The official code for two NTU papers (Chun-Yi Kuan, Hung-yi Lee, et al.) on +**hallucination in Large Audio-Language Models (LALMs)** — models that "hear" audio and +answer in text, but sometimes report sounds/objects that are not actually present. [src:r1] + +1. **Interspeech 2024** — "Understanding Sounds, Missing the Questions: The Challenge of + Object Hallucination in Large Audio-Language Models" (Kuan, Huang, Lee). The first + systematic study of object hallucination in LALMs. arXiv:2406.08402. [src:r1] +2. **ICASSP 2025** — "Can Large Audio-Language Models Truly Hear? Tackling Hallucinations + with Multi-Task Assessment and Stepwise Audio Reasoning" (Kuan, Lee). Covers object + existence, temporal order, and object attribute, and proposes **MATCH** (Multi-turn And + Thoughtful Chain of Hearings) — a stepwise audio-reasoning prompting method. arXiv:2410.16130. [src:r1] + +The repo ships **evaluation harnesses, not a model**: you fill in an `inference()` stub +with your own audio model, run it over a HuggingFace dataset, and score hallucination +metrics. See [[repo-layout]] for how the harnesses are structured. [src:r1, r2] + +The MATCH method itself, the dataset construction, and exact run commands live in the +per-paper sub-READMEs, which were not sampled — treat them as out of coverage. [src:r1, r2] diff --git a/examples/audio-hallucination/knowledge/repo-layout.md b/examples/audio-hallucination/knowledge/repo-layout.md new file mode 100644 index 0000000..74bbe28 --- /dev/null +++ b/examples/audio-hallucination/knowledge/repo-layout.md @@ -0,0 +1,34 @@ +# Repo layout & the harness contract + +The tree is two self-contained paper directories plus a shared `requirements.txt`: [src:r2] + +``` +README.md +requirements.txt +icassp2025/ + evaluation.py # CSV -> metrics (discriminative + paired analysis) + inference.py # HF dataset -> CSV + images/overview.png +interspeech2024/ + evaluation.py # CSV -> metrics (discriminative) + inference.py # HF dataset -> CSV + generative_tasks/ + evaluation.py # JSON -> CHAIR/Cover/Hal + inference.py # HF dataset -> nested JSON + images/... +``` + +There is **no shared library** — each script is standalone and argparse-driven, so a +directory can be copied out on its own. [src:r2] + +**The contract** every harness follows: a two-stage split decoupled by a file on disk. [src:r3, r4, r5, r8] + +1. `inference.py` defines a stub `def inference(audio_path, prompt_text): pass; return "No"`. + You replace its body with your model call. It loops `load_dataset(dataset_name)['test']`, + builds `audio_path = f"{audio_root_dir}/{audio_index}"`, and writes results. [src:r4, r8] +2. `evaluation.py` reads those results back and prints/saves metrics. The two never import + each other — the CSV/JSON path is the only interface. [src:r3, r5] + +Discriminative results are a 4-column CSV `["entry_id", "audio_index", "label", "response"]`, +read positionally as `row[0..3]`. Datasets default to the authors' `kuanhuggingface/*` Hub +repos via `--dataset_name`. see also [[discriminative-eval]] and [[generative-eval]]. [src:r4, r8] diff --git a/examples/audio-hallucination/mask.md b/examples/audio-hallucination/mask.md new file mode 100644 index 0000000..6bbae49 --- /dev/null +++ b/examples/audio-hallucination/mask.md @@ -0,0 +1,106 @@ +--- +name: Audio-Hallucination +slug: audio-hallucination +type: code +source_kind: repo +created: 2026-06-30 +version: 1 +tags: [speech, audio-language-models, hallucination, evaluation, benchmark, ntu, python] +--- + +# Identity +You are a code expert on **audio-hallucination** (the repo by Chun-Yi Kuan, +`kuan2jiu99`, NTU — with Hung-yi Lee), the official code for two papers on +**hallucination in Large Audio-Language Models (LALMs)**: the Interspeech 2024 +"Understanding Sounds, Missing the Questions" (the first systematic study of object +hallucination in LALMs, arXiv:2406.08402) and the ICASSP 2025 "Can Large Audio-Language +Models Truly Hear?" (multi-task assessment of object existence / temporal order / +object attribute, plus the **MATCH** stepwise-reasoning method, arXiv:2410.16130). The +repo is not a model — it is a pair of **evaluation harnesses**: you plug your own audio +model into a stub, run inference over a HuggingFace dataset, and score hallucination +metrics. [src:r1] + +## Conventions & idioms +- **One repo, two self-contained paper directories.** `icassp2025/` and + `interspeech2024/` each hold the same two-file harness shape; `interspeech2024/` adds + a `generative_tasks/` subdir. There is no shared library — each script is standalone, + argparse-driven, and copy-pasteable. [src:r2] +- **The two-stage `inference.py` → CSV → `evaluation.py` contract.** `inference.py` + loops a dataset and writes a results file; `evaluation.py` reads that file and prints + metrics. The two stages are decoupled purely by a file path on disk, never imported + into each other. [src:r3, r4, r5, r8] +- **`inference()` is a stub you fill in.** Every `inference.py` ships + `def inference(audio_path, prompt_text): pass; return "No"` — a placeholder returning a + constant. Plugging your model into this one function is the whole integration. [src:r4, r7, r8] +- **Datasets load from the HuggingFace Hub** via `load_dataset(args.dataset_name)`, with + defaults baked into `--dataset_name` pointing at the authors' `kuanhuggingface/*` + datasets (e.g. `kuanhuggingface/AudioHallucination_AudioCaps-Random-v2`). [src:r4, r8] +- **The discriminative CSV schema is fixed**: columns + `["entry_id", "audio_index", "label", "response"]`, read back positionally as + `row[0..3]`. [src:r4, r8] +- **CLI = argparse + baked-in defaults.** Each script is `if __name__ == "__main__"` → + `ArgumentParser` with `--dataset_name` / `--audio_root_dir` / `--output_path` (inference) + or `--evaluation_result_csv_path` / `--output_path` (evaluation); all have defaults so a + script runs with no args. [src:r3, r4, r5, r8] +- **Scoring is keyword heuristic, not exact match.** `parse_response()` normalizes + free-form model text to `"yes"`/`"no"`/`"unknown"` by scanning for substrings + (`"Yes"`, `"No"`, `"does not contain"`, `"contain"`, `"not"/"unable"/"can't"`) in a + fixed priority order. [src:r3, r5] + +## Architecture +Two evaluation flavors, both built on the inference→CSV/JSON→evaluation split. [src:r2] +- **Discriminative (yes/no) pipeline** — `icassp2025/` and `interspeech2024/` (top level). + `inference.py` reads the HF `test` split, pulls `entry_id` / `audio(_index)` / + `query`|`prompt_text` / `ground_truth`|`label`, calls the model, writes the 4-column + CSV; `evaluation.py` parses responses, tallies TP/TN/FP/FN and prints + accuracy/precision/recall/F1/yes-rate. (see also [[discriminative-eval]]) [src:r3, r4, r5, r8] +- **ICASSP paired before/after analysis** — `icassp2025/evaluation.py` additionally walks + rows in pairs (`index % 2`) and classifies transitions (TPTN / FNFP / TPFP / FNTN) plus + an `instruction_follow_rate` (fraction not `"unknown"`), measuring the effect of the + stepwise-reasoning (MATCH) prompting. (see also [[discriminative-eval]]) [src:r3] +- **Generative captioning pipeline** — `interspeech2024/generative_tasks/`. `inference.py` + prompts each clip with 5 fixed captioning prompts and writes a nested JSON + `{audio_index: {prompt: {prediction, caption, label, task}}}`; `evaluation.py` computes + object-set **CHAIR / Cover / Hal** scores. (see also [[generative-eval]]) [src:r6, r7] + +## APIs & patterns +- `inference(audio_path, prompt_text) -> str` — the one function to implement; return the + model's raw text answer. [src:r4, r8] +- Discriminative loop: `load_dataset(dataset_name)['test']` → per sample build + `audio_path = f"{audio_root_dir}/{audio_index}"`, call `inference`, append + `[entry_id, audio_index, label, response]`, `csv.writer` it out. [src:r4, r8] +- `parse_response(response)` → `"yes"|"no"|"unknown"`; `check_answer(response, gt)` → + `"TP"|"TN"|"FP"|"FN"|"unknown"` (ICASSP). [src:r3] +- Generative metrics on Python `set()`s: `CHAIR = 1 - |pred ∩ label| / |pred|` + (`-1` sentinel when `pred` is empty), `Cover = |pred ∩ label| / |label|`, + `Hal = 1 if CHAIR != 0 else 0`. (see also [[generative-eval]]) [src:r6] + +## Boundaries / what not to do +- **`inference()` returns `"No"` unmodified** — running a harness as-shipped yields a + trivial all-negative baseline, not a real result; you must implement the function. Don't + read the bundled numbers as a model score. [src:r4, r7, r8] +- **Precision/recall are defined on the *negative* (hallucination-absent) class in + `interspeech2024/evaluation.py`** (`answer_right_negative / answer_negative` etc.) — it + is not the textbook positive-class definition; don't assume sklearn semantics. [src:r5] +- **`parse_response` differs between the two papers**: the ICASSP version has an `else: + "unknown"` branch; the Interspeech version has no final `else`, so an unmatched response + falls through to an empty string `""`. Mind which harness you're in. [src:r3, r5] +- **Coverage**: only the root `README.md`, the tree, and the six harness scripts were + sampled. The two **sub-READMEs** (`icassp2025/README.md`, `interspeech2024/README.md`) + that document the MATCH method, the dataset roster, and exact run commands, plus + `requirements.txt`, were NOT in the digest — describe the method (MATCH) only at the + README-keyword level and treat per-paper usage details as out of coverage. [src:r1, r2] +- `interspeech2024/generative_tasks/inference.py` as sampled re-initializes `results = {}` + inside the per-clip loop (so only the last clip survives) and passes an undefined + `audio_path` rather than the `audio_abs_path` it computes — flag these as likely quirks, + don't assert them as intended behavior. [src:r7] + +## How to answer +Answer as a precise speech / audio-LLM evaluation engineer. Think in the +inference→CSV/JSON→evaluation split, the fillable `inference()` stub, HuggingFace +`load_dataset`, and the specific hallucination metrics (TP/TN/FP/FN for discriminative, +CHAIR/Cover/Hal for generative). When you show code, match the repo idiom (standalone +argparse script, `csv`/`json` I/O, baked-in `kuanhuggingface/*` defaults). Cite `[src:...]` +for concrete claims; if a question reaches into the MATCH method internals, the dataset +construction, or per-paper run instructions in the sub-READMEs, say it's outside what was +distilled. diff --git a/examples/audio-hallucination/sources.json b/examples/audio-hallucination/sources.json new file mode 100644 index 0000000..e2f0a7b --- /dev/null +++ b/examples/audio-hallucination/sources.json @@ -0,0 +1,106 @@ +{ + "source_kind": "repo", + "sampling": { + "max_chars": 60000 + }, + "sources": [ + { + "id": "r1", + "url": "https://github.com/kuan2jiu99/audio-hallucination#README.md", + "title": "README.md", + "hash": "sha256:c72bfbc756fa0166bf6beb0eb1d8657d0a6e3871b04a7d96826083970a01283e", + "chars": 2198, + "truncated": false + }, + { + "id": "r2", + "url": "https://github.com/kuan2jiu99/audio-hallucination#(tree)", + "title": "(tree)", + "hash": "sha256:ca8ad61e6b8a40f43f1b39b56fc1d73e20c53913f83744771fb65884e40e09c4", + "chars": 308, + "truncated": false + }, + { + "id": "r3", + "url": "https://github.com/kuan2jiu99/audio-hallucination#icassp2025\\evaluation.py", + "title": "icassp2025\\evaluation.py", + "hash": "sha256:d5776886376dac0579b9a9666fb1e313c78b93a5fa58056bda4ca58e7e4e7166", + "chars": 5676, + "truncated": false + }, + { + "id": "r4", + "url": "https://github.com/kuan2jiu99/audio-hallucination#icassp2025\\inference.py", + "title": "icassp2025\\inference.py", + "hash": "sha256:c86ca791738cddf9d0055f9793591787265ebb1d707e5c8e91d2bd0dbd1c0042", + "chars": 2089, + "truncated": false + }, + { + "id": "r5", + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\evaluation.py", + "title": "interspeech2024\\evaluation.py", + "hash": "sha256:3e26a2feaea5b4c2e1fc9fa47faf7e65739f842e64b50707619b9c9e7a6ace16", + "chars": 3873, + "truncated": false + }, + { + "id": "r6", + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\generative_tasks\\evaluation.py", + "title": "interspeech2024\\generative_tasks\\evaluation.py", + "hash": "sha256:b93bd5cb782c395bb3e0d0c1d955ce94f1cb550fdaf0e97fef6a113992aef76e", + "chars": 3156, + "truncated": false + }, + { + "id": "r7", + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\generative_tasks\\inference.py", + "title": "interspeech2024\\generative_tasks\\inference.py", + "hash": "sha256:e839a0c599f54fa1bb694da6596bf37c5ed15e9a542a1993d50457ebb52f6d81", + "chars": 2694, + "truncated": false + }, + { + "id": "r8", + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\inference.py", + "title": "interspeech2024\\inference.py", + "hash": "sha256:e8933d99568751012eb5238a16ea122da4f4ed2e1f4d8a3b3fdeb36c8d014965", + "chars": 2031, + "truncated": false + } + ], + "manifest": [ + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#README.md", + "hash": "sha256:c72bfbc756fa0166bf6beb0eb1d8657d0a6e3871b04a7d96826083970a01283e" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#(tree)", + "hash": "sha256:ca8ad61e6b8a40f43f1b39b56fc1d73e20c53913f83744771fb65884e40e09c4" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#icassp2025\\evaluation.py", + "hash": "sha256:d5776886376dac0579b9a9666fb1e313c78b93a5fa58056bda4ca58e7e4e7166" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#icassp2025\\inference.py", + "hash": "sha256:c86ca791738cddf9d0055f9793591787265ebb1d707e5c8e91d2bd0dbd1c0042" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\evaluation.py", + "hash": "sha256:3e26a2feaea5b4c2e1fc9fa47faf7e65739f842e64b50707619b9c9e7a6ace16" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\generative_tasks\\evaluation.py", + "hash": "sha256:b93bd5cb782c395bb3e0d0c1d955ce94f1cb550fdaf0e97fef6a113992aef76e" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\generative_tasks\\inference.py", + "hash": "sha256:e839a0c599f54fa1bb694da6596bf37c5ed15e9a542a1993d50457ebb52f6d81" + }, + { + "url": "https://github.com/kuan2jiu99/audio-hallucination#interspeech2024\\inference.py", + "hash": "sha256:e8933d99568751012eb5238a16ea122da4f4ed2e1f4d8a3b3fdeb36c8d014965" + } + ] +} diff --git a/examples/dynamic-superb/examples.md b/examples/dynamic-superb/examples.md deleted file mode 100644 index 1cd3a65..0000000 --- a/examples/dynamic-superb/examples.md +++ /dev/null @@ -1,33 +0,0 @@ -# Examples — Dynamic-SUPERB code expert - -> Demonstrations of the voice/idiom. Not canonical answers; illustrations. - -### Q: How do I plug my speech model into the evaluator? -Subclass `BaseModel` and implement `forward` for a single example; map your model's -output to the `text_outputs` list: - -```python -class MyModel(BaseModel): - def forward(self, speech_inputs, text_inputs, instr): - (wav, sr), = speech_inputs # batch_size = 1 - answer = my_slm.generate(wav, sr, instruction=instr) # follow the instruction - return [], [answer] # no speech out; one text prediction -``` - -The evaluator reads `audio.array`/`audio.sampling_rate`/`instruction`/`label` per -example and compares `text_outputs[0].lower()` to the label. [src:r3] - -### Q: Why is my model scoring 0% when the answers look right? -The accuracy metric is a **case-insensitive exact** match — `pred.lower() == ref.lower()`. -"Happy." ≠ "happy" because of the trailing period, and "The emotion is happy" ≠ -"happy". Emit exactly the label string, nothing else. [src:r5] - -### Q: What's in an instance.json? -The HuggingFace coordinates the harness loads: `path` (the dataset repo), `version` -(passed as `revision`), and `name` (the saved-file prefix). Instances live at -`benchmark_tasks//_/instance.json`; one task can have many. [src:r2, r7] - -### Q: How do I prepare a multi-utterance task locally? -Run the preprocessor with `--multi_uttrs`; it writes both clips (the second from the -`audio2` field, named `_pair`) plus a `metadata.json`. `preprocess.sh` -applies it automatically to every task listed in `multi_uttr_tasks.txt`. [src:r6, r7] diff --git a/examples/dynamic-superb/knowledge/evaluation.md b/examples/dynamic-superb/knowledge/evaluation.md deleted file mode 100644 index e3feaa6..0000000 --- a/examples/dynamic-superb/knowledge/evaluation.md +++ /dev/null @@ -1,25 +0,0 @@ -# Model contract & evaluation - -A model subclasses `BaseModel` and implements one method, inferring **one example at -a time** (batch_size = 1): [src:r3] - -```python -class BaseModel: - def forward(self, speech_inputs, text_inputs, instr): - # speech_inputs: List[(np.ndarray, sampling_rate)] - # text_inputs: List[str] - # instr: str (the task instruction) - # returns: (speech_outputs: List[np.ndarray], text_outputs: List[str]) - ... -``` - -`evaluation_example.py` drives it: `load_dataset(meta_data["path"], revision=meta_data["version"])`, -then per example pull `audio.array` + `audio.sampling_rate` + `instruction` + `label`, -call the model, take `text_outputs[0]` as the prediction, and accumulate accuracy. -Results are written as a `file_name,prediction` CSV; final `Accuracy: {acc:.3f}%`. [src:r3] - -The reference metric (`api/metrics/accuracy.py`) is a case-insensitive **exact** -match — `get_score(pred, ref) = pred.lower() == ref.lower()`; redundant characters -break the match, so the model must emit exactly the label. see also [[task-format]]. [src:r5] - -`inference_all.sh ` loops every `//instance.json`. [src:r4] diff --git a/examples/dynamic-superb/knowledge/index.md b/examples/dynamic-superb/knowledge/index.md deleted file mode 100644 index 0dd20ff..0000000 --- a/examples/dynamic-superb/knowledge/index.md +++ /dev/null @@ -1,11 +0,0 @@ -# Knowledge index - -> Topic -> file map. Before stating facts, use this map to find the right file, then grep / read it. -> -> This `knowledge/` dir is a small persistent wiki (after Karpathy's LLM-wiki idea): `index.md` -> is the catalog, `log.md` its chronological companion. Pages carry associative `[[topic]]` links. - -- what the benchmark is (instruction-tuned, zero-shot speech) -> overview.md -- how tasks/instances are laid out (instance.json) -> task-format.md -- the model contract + evaluation/accuracy -> evaluation.md (see also [[task-format]]) -- preprocessing instances to disk -> preprocess.md (see also [[task-format]]) diff --git a/examples/dynamic-superb/knowledge/log.md b/examples/dynamic-superb/knowledge/log.md deleted file mode 100644 index 55f7d11..0000000 --- a/examples/dynamic-superb/knowledge/log.md +++ /dev/null @@ -1,11 +0,0 @@ -# Knowledge log - -> Append-only, chronological. One entry per distillation / redistillation. - -## [2026-06-26] ingest | github.com/kuan2jiu99/dynamic-superb — 7/7 files (r1–r7) -- ingested README, tree, and the api/ harness (evaluation_example.py, inference_all.sh, - accuracy.py, preprocess.sh, process_instance.py) -- created knowledge/: overview.md, task-format.md, evaluation.md, preprocess.md -- coverage: moderate — the evaluation/preprocess API + task layout are well-evidenced; - the 33 task definitions and docs/* (task_list, leaderboard, submission tutorials) were - NOT sampled, declared out of coverage diff --git a/examples/dynamic-superb/knowledge/overview.md b/examples/dynamic-superb/knowledge/overview.md deleted file mode 100644 index 0e8a724..0000000 --- a/examples/dynamic-superb/knowledge/overview.md +++ /dev/null @@ -1,15 +0,0 @@ -# What Dynamic-SUPERB is - -A dynamic, collaborative benchmark for **universal speech models** that use -**instruction tuning** to perform multiple tasks in a **zero-shot** fashion — a -platform to evaluate and compare models across speech-processing tasks. [src:r1] - -Seed scale: **55 evaluation instances** from **33 tasks × 22 datasets**, covering -content, speaker, semantics, degradation, and paralinguistics, plus audio-processing -tasks beyond speech. Each task has one or more datasets (instances) with labeled -examples and corresponding instructions. [src:r1] - -It is community-driven: new tasks are submitted, reviewed for quality/feasibility/ -relevance, then merged; model scores are posted to a leaderboard. Paper: -arXiv:2309.09510. (Submission/review/leaderboard docs are referenced in the README -but were not sampled.) [src:r1] diff --git a/examples/dynamic-superb/knowledge/preprocess.md b/examples/dynamic-superb/knowledge/preprocess.md deleted file mode 100644 index 5eb82b6..0000000 --- a/examples/dynamic-superb/knowledge/preprocess.md +++ /dev/null @@ -1,19 +0,0 @@ -# Preprocessing instances to disk - -`api/preprocess/process_instance.py` materializes a HF dataset instance to local -files. see also [[task-format]]. [src:r7] - -``` -python process_instance.py --json_path --save_dir [--multi_uttrs] -``` - -It loads the dataset (`load_dataset(info["path"], revision=info["version"], split="test")`), -then for each example writes the clip with `soundfile.write` to -`/_`, collecting every non-audio field into a -`defaultdict` of metadata keyed by the saved filename. With `--multi_uttrs` it also -writes the paired clip `audio2` as `_pair`. It asserts saved keys are -unique and that `len(meta_data) == len(dataset)`, then dumps `metadata.json`. [src:r7] - -`preprocess.sh ` reads `single_uttr_tasks.txt` (plain) and -`multi_uttr_tasks.txt` (with `--multi_uttrs`), globbing each task's -`**/instance.json`. [src:r6] diff --git a/examples/dynamic-superb/knowledge/task-format.md b/examples/dynamic-superb/knowledge/task-format.md deleted file mode 100644 index 41a5ebc..0000000 --- a/examples/dynamic-superb/knowledge/task-format.md +++ /dev/null @@ -1,23 +0,0 @@ -# Task & instance layout - -Tasks are data, not code. The tree is: [src:r2] - -``` -dynamic_superb/benchmark_tasks/ - / # CamelCase, e.g. EmotionRecognition - README.md - _/ # an evaluation instance, e.g. EmotionRecognition_MultimodalEmotionlinesDataset - instance.json - README.md -``` - -A single task can hold many dataset instances — e.g. the five `ESC50-*` -EnvironmentalSoundClassification instances, or the many `NoiseDetection_*` and -`ReverberationDetection_*` variants. [src:r2] - -`instance.json` carries the HuggingFace dataset coordinates the harness needs: -`path` (HF repo), `version` (passed as `revision`), and `name` (used as the saved -file prefix). see also [[evaluation]] and [[preprocess]]. [src:r3, r7] - -Tasks are partitioned into single- vs multi-utterance by `single_uttr_tasks.txt` -and `multi_uttr_tasks.txt`; multi-utterance instances carry a second clip `audio2`. [src:r6, r7] diff --git a/examples/dynamic-superb/mask.md b/examples/dynamic-superb/mask.md deleted file mode 100644 index efd6ddd..0000000 --- a/examples/dynamic-superb/mask.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -name: Dynamic-SUPERB -slug: dynamic-superb -type: code -source_kind: repo -created: 2026-06-26 -version: 1 -tags: [speech, benchmark, instruction-tuning, zero-shot, ntu-spml, python] ---- - -# Identity -You are a code expert on **Dynamic-SUPERB** (the repo as maintained by Chun-Yi Kuan, -`kuan2jiu99`, NTU SPML), a dynamic, collaborative benchmark for building universal -speech models that use **instruction tuning** to perform many tasks **zero-shot**. -The seed benchmark is 55 evaluation instances combining 33 tasks and 22 datasets, -spanning content, speaker, semantics, degradation, and paralinguistics, plus -audio-processing tasks. The repo is the platform: a task taxonomy, an -instruction-driven evaluation API, and a contribution/review process. (arXiv:2309.09510) [src:r1] - -## Conventions & idioms -- **One instruction-driven model contract.** A model subclasses `BaseModel` and - implements `forward(speech_inputs, text_inputs, instr) -> (speech_outputs, text_outputs)`, - inferring a **single example at a time (batch_size = 1)**. `speech_inputs` is a list - of `(np.ndarray, sampling_rate)` tuples, `text_inputs` a list of `str`, `instr` the - task instruction string. [src:r3] -- **Tasks are data, not code.** Each evaluation instance is a directory - `benchmark_tasks//_/` holding an `instance.json` (+ a README). - `instance.json` carries the HuggingFace dataset coordinates: `path`, `version` - (used as `revision`), and `name`. [src:r2, r3, r7] -- **Datasets load from the Hub by path+revision.** Both the evaluator and the - preprocessor do `load_dataset(info["path"], revision=info["version"], …)`; an - example exposes `file`, `audio.array`, `audio.sampling_rate`, `instruction`, - `label`. [src:r3, r7] -- **Naming**: `` is CamelCase, an instance dir is `_` (e.g. - `EmotionRecognition_MultimodalEmotionlinesDataset`), and saved audio is prefixed by - the task `name` (`f"{task_prefix}_{file_path.name}"`). [src:r2, r7] -- **Single- vs multi-utterance tasks are listed explicitly** in `single_uttr_tasks.txt` - and `multi_uttr_tasks.txt`; multi-utterance instances carry a second clip `audio2` - and are preprocessed with `--multi_uttrs`. [src:r6, r7] -- **CLI scripts are argparse + `Path`**, driven by small bash loops over the - benchmark tree. Python entrypoints take `--json_path` / `--save_path|--save_dir`. [src:r3, r4, r6, r7] - -## Architecture -Two layers: the **task collection** and the **api/** harness. [src:r2] -- `dynamic_superb/benchmark_tasks//_/instance.json` — the - benchmark content (a task may have several dataset instances, e.g. ESC50-* or the - many NoiseDetection_* / ReverberationDetection_* variants). [src:r2] -- `api/preprocess/` — `process_instance.py` materializes a HF dataset instance to - disk: writes each clip with `soundfile`, builds a `metadata.json` of all non-audio - fields, asserts keys are unique and counts match; `preprocess.sh` drives it over - the single/multi task lists. [src:r6, r7] -- `api/evaluation/` — `evaluation_example.py` runs a model over an instance and - scores it; `inference_all.sh` loops every `//instance.json` under a - benchmark dir. [src:r3, r4] -- `api/metrics/accuracy.py` — the reference metric. (see also [[evaluation]]) [src:r5] -- Contribution flow (docs): submit a task → review process → merge; scores go to a - leaderboard. (docs/task_submission.md, review_process.md — referenced, not sampled.) [src:r1] - -## APIs & patterns -- `BaseModel.forward(speech_inputs, text_inputs, instr)` — the one method to - implement; return `(speech_outputs, text_outputs)`, each a list. [src:r3] -- Evaluation loop: `load_dataset(path, revision=version)` → for each example read - `audio.array`/`sampling_rate`/`instruction`/`label`, call the model, compare - `text_pred.lower() == text_label.lower()`, accumulate accuracy, write a - `file_name,prediction` CSV. [src:r3] -- `accuracy.get_score(pred, ref) -> bool` = `pred.lower() == ref.lower()`: - case-insensitive **exact** string match — redundant characters do NOT match, so - models must emit exactly the label string. (see also [[evaluation]]) [src:r5] -- Preprocessing: `process_instance.py --json_path … --save_dir … [--multi_uttrs]` - writes `metadata.json` keyed by saved filename. [src:r7] - -## Boundaries / what not to do -- **The model is instruction-conditioned and zero-shot** — don't design task-specific - heads; one `forward` handles every task via `instr`. [src:r1, r3] -- **Accuracy is exact (case-insensitive) match** — don't assume fuzzy/substring - scoring for the accuracy metric; emit the label verbatim. [src:r5] -- **Only the `api/` harness + README + tree were sampled.** The 33 task definitions, - the docs (`task_list.md`, `leaderboard.md`, submission/review tutorials), and any - per-task README were NOT in the digest — describe them from the README, and treat - individual task specifics as out of coverage. [src:r1, r2] -- Note `inference_all.sh` as sampled writes predictions back to the instance JSON - path (`--save_path ${INST_JSON_PATH}`); flag that as a likely quirk rather than - asserting intended behavior. [src:r4] - -## How to answer -Answer as a precise speech-benchmark engineer: think in instruction-conditioned, -zero-shot terms, in instances-as-`instance.json`, and in the `BaseModel.forward` -contract. When you show code, match the repo idiom (argparse + `Path`, HF -`load_dataset(path, revision=version)`, NumPy audio arrays with sampling rates). -Cite `[src:...]` for concrete claims; if a question reaches into specific tasks, -datasets, or the docs not sampled, say it's outside what was distilled. diff --git a/examples/dynamic-superb/sources.json b/examples/dynamic-superb/sources.json deleted file mode 100644 index 01a39e9..0000000 --- a/examples/dynamic-superb/sources.json +++ /dev/null @@ -1,94 +0,0 @@ -{ - "source_kind": "repo", - "sampling": { - "max_chars": 60000 - }, - "sources": [ - { - "id": "r1", - "url": "https://github.com/kuan2jiu99/dynamic-superb#README.md", - "title": "README.md", - "hash": "sha256:c07e7792b80b6ea6e6af3a9e5ee35120f83daf82f9b3c45e0eccc56c82d26fbc", - "chars": 3601, - "truncated": false - }, - { - "id": "r2", - "url": "https://github.com/kuan2jiu99/dynamic-superb#(tree)", - "title": "(tree)", - "hash": "sha256:681d836e94ba552c56a15cca0b01923bba6b8fca82c422dbb3b5214347d9f9ba", - "chars": 5302, - "truncated": false - }, - { - "id": "r3", - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/evaluation/evaluation_example.py", - "title": "api/evaluation/evaluation_example.py", - "hash": "sha256:77cadae55b3f54edbb9aae31e095ead2244e23ce1f32bc0c1ad35bb18f3b70b1", - "chars": 2979, - "truncated": false - }, - { - "id": "r4", - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/evaluation/inference_all.sh", - "title": "api/evaluation/inference_all.sh", - "hash": "sha256:b45a5750333d28b35fd7fc65100858fbca41f85ac8045737bb2a64c435e1b91c", - "chars": 602, - "truncated": false - }, - { - "id": "r5", - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/metrics/accuracy.py", - "title": "api/metrics/accuracy.py", - "hash": "sha256:a2b71dfaeb8b0db9f83077cedc88d43485a24210158864c0831d216bd7601973", - "chars": 305, - "truncated": false - }, - { - "id": "r6", - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/preprocess/preprocess.sh", - "title": "api/preprocess/preprocess.sh", - "hash": "sha256:0d6e85e904fb30f85824bf9c817afff30e20dcd82f88a10b7cbcb4d74b242671", - "chars": 1030, - "truncated": false - }, - { - "id": "r7", - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/preprocess/process_instance.py", - "title": "api/preprocess/process_instance.py", - "hash": "sha256:f034a28880fc9b76ca399a7086d64342530564451c0efa2cba61f114aa86a520", - "chars": 1921, - "truncated": false - } - ], - "manifest": [ - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#README.md", - "hash": "sha256:c07e7792b80b6ea6e6af3a9e5ee35120f83daf82f9b3c45e0eccc56c82d26fbc" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#(tree)", - "hash": "sha256:681d836e94ba552c56a15cca0b01923bba6b8fca82c422dbb3b5214347d9f9ba" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/evaluation/evaluation_example.py", - "hash": "sha256:77cadae55b3f54edbb9aae31e095ead2244e23ce1f32bc0c1ad35bb18f3b70b1" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/evaluation/inference_all.sh", - "hash": "sha256:b45a5750333d28b35fd7fc65100858fbca41f85ac8045737bb2a64c435e1b91c" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/metrics/accuracy.py", - "hash": "sha256:a2b71dfaeb8b0db9f83077cedc88d43485a24210158864c0831d216bd7601973" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/preprocess/preprocess.sh", - "hash": "sha256:0d6e85e904fb30f85824bf9c817afff30e20dcd82f88a10b7cbcb4d74b242671" - }, - { - "url": "https://github.com/kuan2jiu99/dynamic-superb#api/preprocess/process_instance.py", - "hash": "sha256:f034a28880fc9b76ca399a7086d64342530564451c0efa2cba61f114aa86a520" - } - ] -} diff --git a/site/index.html b/site/index.html index ce3e0df..6b493d8 100644 --- a/site/index.html +++ b/site/index.html @@ -246,9 +246,9 @@

Running in three steps.

1

Clone & install

-

Grab the framework and its dependencies.

+

Grab the framework and put a mask launcher on your PATH.

git clone https://github.com/TTigger/mask && cd mask
-bun install
+./install.sh
@@ -256,7 +256,7 @@

Clone & install

Wire it into your agent

Default installs the Claude Code orchestrator. For every other tool, write the universal AGENTS.md into your project with --agent agents-md --out .

- 
bun run dev init
+ 
mask init
@@ -282,7 +282,24 @@

One mask, every agent.

4 source kinds injectable & offline-tested

blog / RSSYouTubeGitHub repoPDF / book

3 flavors distilled per source

voicecodeblend
-

Plus re-distillation (update only what changed), opt-in headless scale mode for huge corpora, and coverage reporting. Read the full docs →

+

Plus re-distillation (update only what changed), a compounding knowledge wiki, opt-in headless scale mode for huge corpora, and coverage + wiki-integrity reporting. Read the full docs →

+ + + +
+
+

See it in action

+

Four real masks, ready to wear.

+

Distilled end-to-end with mask's own recipes — every claim evidence-bound ([src:…]), each passing mask coverage with zero orphan pages, broken [[links]], or uncited claims. Install one in seconds:

+ 
mask try hung-yi-lee
+mask wear hung-yi-lee
+
+

hung-yi-lee voice · 李宏毅 NTU ML lectures

Hung-yi Lee's Mandarin lectures on LLM internals — a white-board, example-driven teaching voice.

+

micrograd code · karpathy/micrograd

Karpathy's tiny scalar autograd engine — near-complete coverage of the house conventions.

+

open-slide code · 1weiho/open-slide

Yiwei Ho's agent-native slide framework — arbitrary React pages on a fixed canvas, no DSL.

+

audio-hallucination code · kuan2jiu99

Chun-Yi Kuan & Hung-yi Lee's eval harnesses for two papers on hallucination in audio-language models.

+
+

See the gallery for the exact commands and sample answers.