Skip to content

chore: Phonemis Build changes after Verbalis migration - #1604

Merged
msluszniak merged 6 commits into
mainfrom
@bo/phonemis-housekeeping
Nov 4, 2026
Merged

msluszniak merged 6 commits into
mainfrom
@bo/phonemis-housekeeping

Conversation

@benITo47

@benITo47 benITo47 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Awaits software-mansion-labs/Phonemis#14. The Phonemis submodule points at that PR's head (8fd520e). If it's squash-merged, this pointer moves to the squash commit before this PR merges.

Summary

Phonemis now preprocesses text (numbers, dates, symbols, abbreviations) with Verbalis 0.1.1, a nested submodule in third-party/common/phonemis/third-party/verbalis. This PR moves the Phonemis submodule to that version and makes the build pick Verbalis up:

  • iOS: the podspec takes Phonemis's source files, excludes, and header search paths from phonemis_sources.rb, which Phonemis now ships.
  • Android: no change. Phonemis's CMake builds Verbalis from the submodule.
  • Checkout: CI (ci.yml, clang-tidy.yml) and third-party/README.md initialize the Phonemis submodule recursively.
  • Submodule URL: now software-mansion-labs/Phonemis, where Phonemis moved.

Tested

  • iOS: the apps/speech simulator build succeeds; 23 Verbalis and 19 Phonemis sources compile, and the app links.
  • Android: the apps/speech debug APK builds;
  • C++ host tests (run-native-tests.sh): 195 / 195 pass, including PhonemizerTest 16 / 16.
  • JS tests: 5,018 / 5,018 pass, the podspec tests included.

Blast Radius: speech.

  • Phonemes change wherever Verbalis reads text differently than Phonemis's old preprocessor: mostly numbers, dates, and abbreviations. Against reviewed readings, the new reading is right in 3,869 corpus sentences where the old was wrong, and wrong in 22 where the old was right.
  • Local clones need git submodule update --init --recursive. Without the nested submodule, Android's CMake downloads Verbalis and the iOS podspec can't find it

Phonemis now preprocesses text with Verbalis, a nested submodule in
third-party/verbalis. The podspec takes Phonemis's sources and header search paths
from its phonemis_sources.rb, so they include Verbalis and follow Phonemis's layout.
CI and the third-party README check the submodule out recursively; Android needs no
change, since Phonemis's CMake builds Verbalis from the submodule. The submodule URL
follows Phonemis's move to software-mansion-labs.

Phonemis now includes <optional> and <string> where it uses them, so the host
tests no longer force-include them for libstdc++.

@msluszniak msluszniak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also outside the diff:

  • compile_flags.txt needs -isystemthird-party/common/phonemis/third-party/verbalis/cpp/include: phonemizer.h now reaches <verbalis/verbalizer.h> via pipeline.h, so clang-tidy/clangd can't parse it.
  • Root LICENSE: add Verbalis (MIT) to the third-party components; cn2an code moved there from Phonemis.
  • cpp/tests/README.md:65 still lacks --recursive.

Comment thread .github/workflows/clang-tidy.yml Resolved
@msluszniak msluszniak added 3rd party package Issue related to 3rd party packages, but not ExecuTorch, e.g. Expo chore PRs that are chores labels Nov 4, 2026
@msluszniak msluszniak changed the title Phonemis Build changes after Verbalis migration chore: Phonemis Build changes after Verbalis migration Nov 4, 2026
- compile_flags.txt: add Verbalis's include path, so clangd and clang-tidy
  parse the speech sources, which reach <verbalis/verbalizer.h> through
  Phonemis's headers.
- clang-tidy.yml: check out only Phonemis and Verbalis, as ci.yml does,
  instead of every submodule.
- ci.yml: check out Phonemis in the TypeScript API tests job, since the
  podspec they evaluate requires phonemis_sources.rb.
- create-package.sh: pack only Verbalis's cpp/ and license files.
- LICENSE: list Verbalis (MIT), which bundles the cn2an port.
- cpp/tests/README.md: check out Phonemis recursively.
…ckage

Phonemis main has the merged Verbalis work, and notices that credit Verbalis.
create-package.sh also leaves out Phonemis's README image and README source,
and restores the whole submodule after packing.
@msluszniak

Copy link
Copy Markdown
Member

Please bump phonemis to v0.4.0 (5ec260e). Code matches the current pin, but the pin's THIRD_PARTY_NOTICES.md doesn't list Verbalis yet, and the new LICENSE text points there. The bump also adds readme.source.md and assets/ to the tarball, so trim them in create-package.sh.

@benITo47

benITo47 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

I think you wrote that before the commit landed :-p

Done in 2b6849f: phonemis is at 5ec260e (v0.4.0), so its THIRD_PARTY_NOTICES.md now lists Verbalis. create-package.sh also trims readme.source.md and assets/, and the restore step now restores the whole submodule.

@benITo47
benITo47 marked this pull request as ready for review November 4, 2026 13:32
@msluszniak
msluszniak self-requested a review November 4, 2026 13:33
PhonemizerOutputTest runs the en-us pipeline on the shipped tagger and
lexicon, fetched as fixtures, and pins the phonemes and word locations for
numbers, money, dates, abbreviations, times and units.
PhonemizerOutputTest adds en-gb and zh on their shipped data.
PreprocessingTest pins what phonemis feeds the neural model for de, fr,
es, it, pl, pt and hi. Known misreadings point at Verbalis#8.
A phonemis or Verbalis bump that changes a reading now fails with the old
and new value and is accepted with RNE_UPDATE_SNAPSHOTS=1, so the PR shows
the diff. LeavesNoDigitsOrSymbols and LocatesEveryWordOfTheInput hold
across versions and catch real regressions.
@msluszniak
msluszniak merged commit 8187c90 into main Nov 4, 2026
7 checks passed
@msluszniak
msluszniak deleted the @bo/phonemis-housekeeping branch November 4, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3rd party package Issue related to 3rd party packages, but not ExecuTorch, e.g. Expo chore PRs that are chores

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants