Granary v2 postprocessing

[nune-tadevosyan]

Description

Adds a production-ready postprocessing pipeline for Granary v2 ASR manifests under tutorials/audio/granary_v2_postprocessing/.

The pipeline reads JSONL manifests produced by ASR inference, applies text cleaning and quality filtering, and writes output manifests preserving the full input directory structure. All
entries are kept in the output — low-quality entries are flagged with skip_me=1 for downstream use rather than dropped.

Pipeline stages
ALMManifestReader - Reads JSONL — one AudioTask per line
InitializeFieldsStage - Copies pred_text → cleaned_text; sets skip_me = 0
RegexSubstitutionStage - Normalizes cleaned_text (quotes, dashes, brackets, whitespace)
WhisperHallucinationStage - Sets skip_me = 1 for repeated n-grams, long words, known hallucination phrases, or abnormal chars/duration rate
FastTextLIDStage - Sets skip_me = 1 for non-English or low-confidence language ID
FinalizeFieldsStage - Renames text → v1_text, promotes cleaned_text → text, drops pnc/itn/timestamp
ALMManifestWriterStage - Writes all entries (clean and flagged) to mirrored output paths

Key features:

• Directory-based input — recursively finds all *.jsonl under --input_dir, no YAML config needed
• Checkpointing — manifests with an existing non-empty output are skipped on rerun; atomic writes (.tmp + rename) prevent partial files from being mistaken for complete output
• Slurm-ready — submit.sh groups manifests into configurable chunks and submits one job per chunk, all running in parallel

  bash tutorials/audio/granary_v2_postprocessing/submit.sh \
      /path/to/output_root \
      /path/to/input_dir

Important to set how many manifests should be processed during one job submission

[nithinraok]

what does low-confidence transcriptions predictions mean here?

[nune-tadevosyan]

FastText gives two predicitions language and probability of that language.

1. If language if anything different than English we skip it.
2. If language is English but with probability being smaller than 0.7 we again skip it

[nithinraok]

Were you able to listen to some samples where it predicted english but with lower probablity, curious of such data from our Granary set