feat: add Malayalam (ml) as a target language — research, glossary strategy & model recommendation

[mmcky]

Summary

A native Malayalam-speaking contributor — a financial-literacy educator in Kerala doing non-commercial, pro-bono educational work — has asked to adapt QuantEcon programming and economics material for Malayalam learners. This issue captures the research we did into whether and how we can add Malayalam (ml) as a target language in action-translation, and recommends how to best support it (model tier, glossary strategy, and the role of Indic-specialist models). Mechanically, adding the language is the same path we already used for French (#68) and Japanese (#69); the substance below is about getting the quality right for a genuinely under-resourced language pair (en→ml).

TL;DR — 🟡 feasible and worthwhile, with guardrails. en→ml hits a real resource cliff in the direction this tool runs in — and the scarcity is in the training data and tooling, not the language. Ship it as a human-reviewed draft pipeline, default ml to Opus rather than the global Sonnet, and redesign the glossary with a three-way translate/transliterate/keep-English policy instead of cloning the zh-cn/fa pattern.

A note on terminology. Throughout, "under-resourced" refers narrowly to the availability of digital and NLP resources — parallel corpora, benchmarks, term banks, and pretrained models — not to the language itself. Malayalam is a classical language of India with ~38 million speakers and a deep literary and scholarly tradition. The gap this issue addresses is in the tooling and training data the AI industry has invested in it so far — and the point of this work is to help close that gap.

---

Research findings (frontloaded)

1. Translation-quality landscape

The dominant signal across every Indic benchmark is the well-resourced vs under-resourced gap, and Malayalam sits well below the best-resourced Indic languages. Crucially, the weak direction is exactly ours: generating Malayalam script and morphology (en→ml) is far harder than reading it (ml→en).

• The en→ml generation direction collapses for general LLMs. On IndicGenBench (FLORES-IN, chrF, one-shot), GPT-4's en→ml score was 28.4 vs PaLM-2-L's 59.7 and BLOOMZ's 66.2 — roughly half. ml→en is much closer (GPT-4 58.6 vs PaLM-2 68.0). This tool runs en→ml, so this is the relevant cliff.
• Understanding does not imply generation. GPT-4o leads Malayalam comprehension on MILU at ~71%, but that is still ~10 points below English and ~6 below Hindi — and comprehension ≠ writing fluent, correct Malayalam.
• Dedicated NMT beats general LLMs on generation by a wide margin. On FLORES-200, GPT-3.5 ml→en chrF was 50.10 vs IndicTrans2 66.20 and Google 66.40 — a ~16 chrF gap even in the easier direction.
• Claude-on-Malayalam is unmeasured. Anthropic publishes no per-language Malayalam translation scores. Its only public multilingual table is 14 languages, MMLU understanding relative to English, and Malayalam is absent — the only Indic entries are the two best-resourced (Hindi 96.7%, Bengali 95.4%, on Sonnet 4.5). Claude's strong MT reputation (WMT24 wins, blind professional ratings) is built entirely on high-resource European/East-Asian pairs. Confidence that this transfers to Malayalam is low and should be communicated that way to the contributor.
• Failure modes are intrinsic, not transient: agglutinative morphology, complex sandhi (word-joining), long compounds, native-script generation, and English code-mixing. Our content adds a second axis of difficulty — dense quantitative-economics and Python prose, a domain Malayalam corpora barely cover.
• Scaling helps the data-scarce floor more than the high-resource ceiling, but does not close the gap. Relative gains from larger models are largest exactly where training data is thinnest, but no frontier model exceeds ~58% average on the hardest Indic languages. Bigger models reduce post-edit burden; they do not eliminate it.

2. Glossary strategy — the single most important finding

Real Kerala STEM and finance practice is a three-way split driven by how established each concept is — not the uniform full-translation pattern our zh-cn and fa glossaries use (those translate even acronyms, e.g. GDP → 国内生产总值 / تولید ناخالص داخلی). Forcing a native coinage for every imported term would produce technically-correct-but-unread Malayalam — the exact register gap that exists between official term banks and actual classroom/Wikipedia usage. Kerala STEM education genuinely code-mixes English terms ("Manglish" is a real, prevalent register, not sloppiness).

Category (by context)	Treatment	Example
Established economics / finance / macro-micro concepts	translate	inflation → പണപ്പെരുപ്പം, demand → ചോദനം, interest rate → പലിശനിരക്ക്
Established core-math concepts	translate	determinant → സാരണികം, transpose → പക്ഷാന്തരിതം, row/column → നിര/വരി
Imported / computing / named-method terms	transliterate (Malayalam script)	matrix → മാട്രിക്സ്, function → ഫങ്ഷൻ, algorithm → അൽഗൊരിതം
Proper names (economists, mathematicians; ~40 entries)	transliterate	Lionel Robbins → ലൊയ്നൽ റോബിൻസ് (same policy fa/zh already apply)
Acronyms	native expansion as the ml value + transliterated short form	GDP → മൊത്ത ആഭ്യന്തര ഉത്പാദനം (ജി.ഡി.പി)
Code tokens, library/package names, file formats, keywords	keep-english (Latin)	numpy, pandas, def, for, import, AR(1)

3. Indic-specialist models (IndicTrans2 / Sarvam) — auxiliary, never in the pipeline

The tool is Claude-only and structure-preserving by design, and that constraint is correct. IndicTrans2 (AI4Bharat, MIT-licensed, covers Malayalam) and similar are plain-sentence MT systems with no notion of MyST structure, code fences, math, or directives — wiring them into the pipeline would break the document-preservation guarantees that are the whole point of the tool. They do add value offline, at authoring time: seeding the ml glossary draft, cross-checking each term against a Claude candidate (agreement → high confidence, divergence → flag for the human reviewer), and optionally back-translating ml→en as a QA spot-check. Treat them as a way to raise the draft floor and route reviewer attention — not as ground truth.

4. Community Malayalam LLMs and datasets (assessed, not recommended as the translator)

• MalayaLLM (LLaMA-2 7B / Gemma-2 9B community fine-tune): not viable as the translator — too small for faithful technical translation, no structure handling, no published benchmarks, and self-hosting would require re-architecting the Claude-only pipeline. At most a reviewer-side second opinion.
• Parallel corpora (Samanantar ~49.7M pairs, BPCC ~230M): mainly useful as glossary-mining sources; their value is largely already baked into IndicTrans2. We are not training a model, so raw scale is low-value to us.
• University of Surrey / TechLiebe en→ml dataset (~8k segments, finance/legal/news, with human scores, post-edits, and an error taxonomy): the best-matched single resource for the contributor's finance focus — useful for glossary seeding, few-shot exemplars, and as a ready-made review rubric. Public release was slated for April 2026; confirm availability and license before use.

Verification note

The translation-quality claims above were independently fact-checked against primary sources (Anthropic docs, MILU/IndicGenBench/FLORES papers). One MILU framing was downgraded from "supported" to "overstated" (it over-attributed Malayalam's score to agglutinative morphology, which the paper names only for Tamil/Telugu). The Claude-on-Malayalam gap and the en→ml weakness are well-supported; the absolute Claude-ml quality remains an inference, not a measurement.

Sources

• Anthropic multilingual support docs (14-language MMLU-understanding table; Malayalam absent)
• MILU — Multi-task Indic Language Understanding, NAACL 2025 (arXiv 2411.02538)
• IndicGenBench, ACL 2024 (arXiv 2404.16816)
• "Assessing Translation capabilities of LLMs…", EAMT 2024 (arXiv 2311.09216)
• IndicTrans2 (arXiv 2305.16307; https://github.com/AI4Bharat/IndicTrans2)
• Analysis of Indic capabilities / IndicParam scaling (arXiv 2501.13912, 2512.00333)
• Malayalam Wikipedia term conventions; Wiktionary "About Malayalam"; Kerala Bhasha Institute / CSTT term banks; fincash Malayalam financial-literacy material
• MalayaLLM (https://github.com/VishnuPJ/MalayaLLM); Surrey/TechLiebe en→ml dataset (TechXplore, Jan 2026)

---

How we might best support Malayalam in this tool

Model: per-language Opus override

Default ml to Opus (claude-opus-4-8) rather than inheriting the global claude-sonnet-4-6. The cost premium (~1.7×) is negligible at one contributor's volume, scaling buys the most where training data is thinnest, and the dominant cost here is the contributor's review time — which better raw output directly reduces. Keep Sonnet for the proven zh-cn/fa. This needs:

• A per-language model override (model is currently a single global input in src/inputs.ts / action.yml), and
• Adding claude-opus-4-8 to the VALID_MODEL_PATTERNS allowlist in src/inputs.ts (today it only warns past claude-opus-4-6; the warning is non-blocking but should be cleaned up).

Glossary: redesign, don't clone

Add an ml-specific per-term treatment field (translate | transliterate | keep-english) to the glossary objects, defaulted by context, applying the three-way split in the table above. This is an additive change to the existing {en, <lang>, context} schema; zh-cn/fa keep their current shape. Land glossary/ml.json as a 🟡 machine draft via the established draft-then-native-review workflow, then route to the contributor for review before promoting to ✅.

Process: native review is the quality gate

For a language this under-served by existing MT resources, with zero published Claude baseline, native-speaker review is not a nicety — it is the quality bar. Wire the contributor's review in as a required step, treat the first lectures as a calibration batch, and use reviewer edit-effort as our de-facto quality metric (since none is published). Feed their corrections back into the glossary.

Keep the pipeline Claude-only and structure-safe

Use IndicTrans2 offline to seed/cross-check the glossary; never in the live path. Add a lint check that Latin code tokens, package names, and file formats survive verbatim — the morphological-correctness pressure on Malayalam makes leakage likelier than for fa/zh.

---

Proposed implementation checklist

• Add an 'ml' entry to LANGUAGE_CONFIGS in src/language-config.ts (Malayalam punctuation/register rules)
• Add treatment field support for ml glossary terms (schema + translator prompt wiring)
• Generate glossary/ml.json as a machine draft (357-term set) using the three-way treatment policy; seed/cross-check with IndicTrans2
• Add a per-language model override and default ml → claude-opus-4-8; add it to the model allowlist
• npm run build to bundle dist-action/glossary/ml.json (so the dist-freshness CI guard passes and the loader auto-resolves it)
• Add a code/structure lint (Latin code tokens survive verbatim)
• Tests + docs (docs/user/language-config.md, docs/user/glossary.md, glossary/README.md)
• Route the draft glossary + first lectures to the native-speaker contributor for review (required gate)

Open questions for maintainers

• Is a per-language model override worth adding now, or do we ship ml on a globally-overridden Opus run for the first batch and generalize later?
• Do we want the treatment field to be ml-only, or is it worth generalizing the glossary schema across languages?
• Should we wait on the Surrey en→ml dataset (license TBD) before drafting the glossary, or draft now and reconcile later?

[mmcky]

Update: native-speaker review collapses the three-way split → keep-English-dominant

Our Malayalam contributor and reviewer, Adisankar Manoj Thanuja (Kerala finance educator), reviewed this issue and proposed a significant simplification of the glossary policy, backed by 16 worked examples across economics, finance, statistics, and programming. On register judgment for a low-resource language with no Malayalam-medium STEM infrastructure, the native-speaker educator outranks the corpus-derived inference, and we're adopting it.

The core change: for Malayalam specifically, keep technical terms in English and translate only the connective Malayalam prose around them. Kerala learners use English technical terminology natively from earliest exposure (there is no Malayalam-language university teaching economics/CS and no active classroom term bank), so translating or transliterating technical terms reads archaic — "government gazette," not classroom. This sharpens, rather than contradicts, the original research finding about Kerala's code-mixed ("Manglish") register.

Revised policy (supersedes the table above)

Category	Original (research)	Adopted (native review)
Technical terms — economics, finance, statistics, programming	translate / transliterate by sub-type	keep English (dominant)
Transliteration	a category	effectively dropped
Translate	established concepts	only everyday non-technical words (country → രാജ്യം, year → വർഷം)
Acronyms (GDP)	native expansion + ജി.ഡി.പി	keep English ("GDP")
Proper names (Solow, Robbins)	transliterate	keep English/Latin (proposed)

The characteristic pattern is Malayalam grammar wrapping English roots inline: economy-യിലെ, bond-ന്റെ, asset classes-ൽ, and English verbs with a Malayalam auxiliary (process ചെയ്ത്, return ചെയ്യുന്നു). This English-root-plus-Malayalam-inflection task is an LLM strength that a sentence-level NMT system can't do — it reinforces the Claude-only + per-language Opus recommendation.

What this changes in the tool design

• The lever moves from the glossary to the language-config prompt rules. The hard behavior is "keep the English term, inflect Malayalam around it" — that lives in src/language-config.ts, not a 357-row table.
• Glossary v1 needs no schema change. Use ml == en for technical terms (tells Claude "GDP stays GDP") and a Malayalam value only for everyday words. This defers the proposed treatment field — we add it later only if the calibration data shows we need it.
• The primary risk flips from over-translation to consistency — a term appearing 20× must be handled identically each time (Adisankar's explicit requirement).

Drafted artifacts (local, uncommitted — for the calibration run)

1. ml entry in src/language-config.ts encoding the keep-English rules:

'ml': {
  code: 'ml',
  name: 'Malayalam',
  additionalRules: [
    'Keep ALL technical and domain terms in their original English/Latin form — do NOT translate or transliterate them into Malayalam script. This covers economics (inflation, GDP, recession, interest rate, demand, supply), finance (equity, bond, yield, portfolio, asset class), statistics (regression, correlation, standard deviation, normal distribution), mathematics, and programming (function, loop, library, variable, dataset, numpy, pandas), plus acronyms (GDP, RBI) and named institutions (Federal Reserve)',
    'Translate only the surrounding Malayalam prose and common NON-technical everyday words that have a natural, in-use Malayalam equivalent (e.g. country → രാജ്യം, year → വർഷം, increase → ഉയർത്തുക, before → മുമ്പ്)',
    'Attach Malayalam case-suffixes, postpositions, and sandhi directly to the English term, hyphenated where natural (e.g. economy-യിലെ, bond-ന്റെ, asset classes-ൽ, consumers-ന്റെ); for English verbs, use the English verb plus a Malayalam auxiliary (e.g. process ചെയ്ത്, return ചെയ്യുന്നു, execute ചെയ്യുന്നു)',
    'Keep proper names (economists, researchers, institutions) in English/Latin script — do not transliterate them',
    'Handle every term consistently across the whole document — a term kept in English must remain English on every occurrence, and a word translated to Malayalam must use the same Malayalam rendering each time',
    'Use a natural classroom/educational register, not a formal government-gazette style; where a Malayalam word genuinely aids comprehension for a borderline non-technical concept, it may be given with the English in parentheses on first use (e.g. ബന്ധം (relationship)) — use sparingly',
  ],
},

2. Seed glossary/ml.json (60-term draft demonstrating the policy): ~47 technical terms pinned to English (ml == en) and 13 everyday words translated, with every Malayalam value taken directly from Adisankar's worked examples. The terms most at risk of over-translation are the common English words that also have Malayalam equivalents (demand, supply, value, price, risk, returns) — pinning those to English is the glossary's real job here.

Decisions to confirm (proposed defaults)

• Proper names → keep English/Latin (Adisankar's doc didn't cover names; his logic extends cleanly). @-reviewer to confirm.
• Acronyms → keep English ("GDP", not ജി.ഡി.പി) — confirmed by his examples.
• Borderline non-technical terms → Malayalam with English in parens on first use (ബന്ധം (relationship)), used sparingly.
• v1 = zero-schema-change (ml == en + language-config rule); defer the treatment field.

Updated next step

Run a calibration batch of real lectures on this config (Opus, ml), and Adisankar flags each over-translation instance so we tune the glossary/rules from real output rather than more theory. This replaces the speculative parts of the implementation checklist above.

Thanks to Adisankar for the detailed, example-driven review — it materially improved the plan.