Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions articles/EDITORIAL_NOTES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# GlyphFlow article notes

These notes record the editorial research behind the Zenn and DEV Community
drafts in this directory. They are deliberately kept with the drafts so that a
future revision does not turn them into release announcements.

## What the research changed

The articles are not translations of each other.

| Draft | Question it answers | Reader's outcome |
| ---------------------------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| `zenn-web-speech-caption-ui-ja.md` | How can a browser-only caption move inside a word when the browser only reports word boundaries? | Understand the event model, the CSS rendering choice, and the limits. |
| `devto-speech-captions-en.md` | Why use one gradient-clipped word instead of changing a whole word or splitting it into letters? | Understand the UI trade-off and the bounded between-event animation. |

Both drafts lead with a concrete mismatch between an expected UI and a first,
simpler implementation. They retain the project's real constraints rather than
claiming precise character or phoneme timing.

## Research sources and publishing constraints

- [Zenn community guidelines](https://zenn.dev/guideline): Zenn values concrete
experience and trial-and-error, asks authors to avoid advertising as the main
purpose, and recommends a clear overview, audience, context, and reproducible
conditions.
- [Zenn: writing a better technical article](https://zenn.dev/northward/articles/how-to-write-better-tech-articles): check that the title matches the article,
name the environment, include enough detail to reproduce the result, and read
the finished piece aloud for awkward rhythm.
- [Zenn: using AI support while writing](https://zenn.dev/rin/articles/bdbdab57997e1e) and
[a personal AI-writing workflow](https://zenn.dev/yoshiko/articles/my-ai-writing): use AI for structure or review, but rewrite the final voice from actual
experience rather than accepting generic prose.
- [DEV: high-quality posts](https://dev.to/devteam/how-to-write-a-high-quality-post-on-dev-3me0): use accurate titles, correct tags, alt text for images, and
deliver what the title promises.
- [DEV: AI-assisted article guidelines](https://dev.to/guidelines-for-ai-assisted-articles-on-dev): AI assistance must be disclosed. The author must stand
behind factual accuracy, and an article must not be created mainly to drive
external traffic or backlinks.
- [MDN: `SpeechSynthesisUtterance` `boundary`](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance/boundary_event): a boundary event
marks a word or sentence boundary. It is not a per-character timing API and is
not Baseline across widely used browsers.

## Voice rules applied to these drafts

1. Start from an observed mismatch, not a product definition. In this project,
changing an entire word's color did not convey the motion the UI needed.
2. Preserve decisions that another author could not make generically: one word
span, gradient clipping, `charIndex` mapping, bounded progress, and no audio
alignment service.
3. Include the trade-offs next to the technique. A boundary event starts a word;
it does not reveal when every letter is spoken.
4. Do not manufacture a founder story, performance result, compatibility claim,
or user outcome. Before publication, replace any first-person wording with
the author's own, factually correct memory of the work.
5. Delete generic sales language. In particular, avoid opening with phrases such
as "a library that enables", ending with a broad request for feedback, or
presenting a symmetric feature list as the substance of the article.
6. Vary paragraph length only where it follows the thought. Artificially clipped
fragments and deliberate typos are not a human voice.
7. Use a small, verified code sample. Link to the repository for the complete
implementation instead of pasting a large, context-free excerpt.

## Pre-publication checklist

- [ ] The author has replaced or approved every first-person statement.
- [ ] Browser, OS, voice, rate, and language claims match a manual check.
- [ ] Code samples have been copied from, or tested against, the release source.
- [ ] Screenshots/GIFs show the actual inside-glyph wipe; no mock UI is presented
as a live boundary-timed result.
- [ ] The final call to action is a short project-links section, not the article's
purpose.
- [ ] The DEV post includes an AI-assistance disclosure if this draft materially
shaped the final prose, as DEV requires.
- [ ] The author reads the Japanese and English drafts aloud and removes wording
they would not naturally use.

## Project facts verified from this repository

- Package: `@enumura/glyphflow`, version `1.0.0`.
- Runtime package contents: `dist`, `README.md`, and `LICENSE` only.
- Rendering keeps each word as one `.kn-word` element and changes
`--kn-progress`; it does not split a word into character elements.
- `boundary` event handling uses `charIndex`, ignores stale/duplicate/out-of-order
events, and records a measured duration only when a following boundary is
received.
- The visual predictor uses lightweight spelling costs and bounded progress. It
targets 94% by its predicted end and never moves beyond 98.5% until the next
event or completion resolves the word.
- The documented initial support target is Chrome Desktop, English text, and a
voice that emits usable word boundaries.
138 changes: 138 additions & 0 deletions articles/devto-speech-captions-en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
<!--
DEV Community draft

Suggested tags: webdev, javascript, typescript, css
Do not use this as a backlink-only announcement.

Disclosure required by DEV if this draft materially shapes the published prose:
"Disclosure: I used AI assistance to organize and edit this article. I verified
the technical details against the linked source code and documentation."
-->

# I wanted browser speech captions to move, not blink

The first version of this UI only changed the color of the current word.

That did answer one question: which word is the browser speaking? It did not
look like the caption was moving with the voice. A word would sit unchanged and
then switch all at once. For a read-along interface, I wanted the color boundary
to travel through the letters instead.

I built [GlyphFlow](https://github.com/speech-caption/glyphflow) around that
constraint. It is a small TypeScript library for Web Speech captions. This post
is about the rendering and timing decisions behind it, not a claim that the
browser can provide character-level speech alignment.

## The useful event is also an incomplete one

`SpeechSynthesisUtterance` can emit a `boundary` event while an utterance is
spoken. The event includes a character index and elapsed time.

```ts
utterance.onboundary = (event) => {
console.log(event.name, event.charIndex, event.elapsedTime);
};
```

That is enough to identify a word start in the original input. It is not enough
to say when the voice reached the second `t` in `letter`.

This distinction shaped the whole implementation. The browser is the source of
truth for word starts. The motion between word starts is a visual estimate.

It also puts a real limit on the project: Web Speech behavior depends on the
browser, OS, and installed voice. The initial support target is Chrome Desktop,
English text, and voices that produce usable word-boundary events. MDN marks
the event as limited availability rather than a Baseline feature. [Its boundary
event documentation](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance/boundary_event)
is worth reading before treating it as a universal captioning API.

## One word, one element

I did not split each word into character elements. That would make it easy to
light up letters one at a time, but that is a different visual language: a
sequence of discrete changes rather than one boundary moving through a glyph.

The rendered word stays as one span.

```html
<span class="kn-word" style="--kn-progress: 42%">caption</span>
```

The CSS background is the moving part.

```css
.kn-word {
color: transparent;
background: linear-gradient(
90deg,
var(--kn-highlight-color, #f3f7ff) 0%,
var(--kn-highlight-color, #8da7ff) var(--kn-progress),
var(--kn-unhighlight-color, #475061) var(--kn-progress),
var(--kn-unhighlight-color, #475061) 100%
);
background-clip: text;
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
```

Updating `--kn-progress` from `0%` to `100%` moves the transition through the
actual letter shapes. The DOM still represents a word as a word, and punctuation
and whitespace can remain faithful to the source text.

## A boundary starts the motion; the next one measures it

There is a quiet gap in this model. A boundary tells me a word has started, but
the next boundary has not arrived yet. Leaving the word static for that whole
gap looks broken, so GlyphFlow starts a `requestAnimationFrame` loop.

The initial duration estimate is intentionally lightweight: word length, a few
letter patterns, speech rate, and nearby punctuation. There is no dictionary,
phoneme model, audio file, or network call involved.

The estimate does not complete the word at exactly 100%. It reaches 94% at the
predicted end, then eases no farther than 98.5%. When the next word boundary
arrives, the previous word becomes 100% and the elapsed-time difference becomes
an observed duration.

```ts
const actualDuration = nextBoundary.elapsedTime - currentBoundary.elapsedTime;
```

That observed duration can gently influence later estimates in the same
utterance. It is useful when a particular voice and rate are consistently faster
or slower than the initial guess. It is not phoneme alignment, and a final word
has no following boundary from which to derive an observed duration.

## Browser events need defensive state handling

The more interesting bugs were not in the gradient. They were in event timing.

A word boundary can be duplicated, arrive out of order, point near rather than
at a token start, or arrive after a narration was cancelled. A paused caption
must not keep progressing just because an animation frame is still scheduled.

GlyphFlow assigns each `speak()` call a session identity. Events from an older
utterance cannot update a newer session's DOM or state. Duplicate and stale
boundaries are ignored. Pause time is excluded from visual progress. If speech
finishes without usable word boundaries, the caller receives an `unsupported`
state instead of a caption that quietly pretends it was synchronized.

Those cases are less impressive in a demo than a flowing gradient, but they are
the difference between a visual experiment and a component that can be embedded
in another page.

## What this is, and what it is not

The result is useful when a browser voice should give a reader a place to look:
read-along captions, language exercises, or a narrated explanation. It is not a
tool for synchronizing supplied audio, producing MP3 files, or deriving
per-letter pronunciation timestamps.

The project is MIT-licensed. If you want to inspect the implementation or try
the demo:

- Source: https://github.com/speech-caption/glyphflow
- Package: https://www.npmjs.com/package/@enumura/glyphflow
- Demo: https://speech-caption.github.io/glyphflow/
134 changes: 134 additions & 0 deletions articles/zenn-web-speech-caption-ui-ja.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
title: "Web Speech APIの単語境界だけで、文字の内側を塗り進める字幕UIを作る"
emoji: "🔊"
type: "tech"
topics: ["typescript", "webapi", "css", "webspeechapi"]
published: false
---

## はじめに

ブラウザの読み上げに合わせて字幕を動かすなら、今読んでいる単語だけ色を変えればよいと思っていました。

実際に表示してみると、それでは足りませんでした。単語の切り替わりは分かりますが、音声と文字が一緒に進んでいる感じが出ません。欲しかったのは、単語が一瞬で明るくなる表示ではなく、文字の形の中を左から右へ色が進む表示です。

この記事では、Web Speech APIが返す単語境界イベントだけを使い、音声ファイルや強制アラインメントなしでその表示を作るまでの設計をまとめます。完成コードは [GlyphFlow](https://github.com/speech-caption/glyphflow) にあります。

## この記事で扱う範囲

対象は、Chrome Desktopで英語テキストを読み上げ、単語境界を返すvoiceです。Web Speech APIの挙動はブラウザ、OS、voiceで変わります。特に`boundary`イベントは広く一律に使えるAPIではありません。 [MDN](https://developer.mozilla.org/ja/docs/Web/API/SpeechSynthesisUtterance/boundary_event)

この記事は、文字や音素ごとの正確な発音時刻を求めるものではありません。そこまで必要なら、音声ファイルとアラインメント情報を使う別の問題になります。

## まず分かるのは「単語の開始」だけ

`SpeechSynthesisUtterance` の `boundary` イベントには、発話テキスト内の位置である `charIndex` と、発話開始からの `elapsedTime` が入ります。

```ts
utterance.onboundary = (event) => {
console.log(event.name, event.charIndex, event.elapsedTime);
};
```

ここで重要なのは、イベントがアニメーションのフレームではないことです。`charIndex` が教えてくれるのは「このあたりの単語が始まった」であって、「この文字を発音した」ではありません。

そのため、元のテキストを先に単語化し、各単語の開始・終了位置を保存します。句読点や連続する空白を壊さずに表示したいので、単語そのものと、その後ろに続く文字列を分けて保持します。

```ts
type WordToken = {
text: string;
start: number;
end: number;
trailing: string;
units: number;
};
```

同じ単語が続く文でも、単語の文字列ではなく`charIndex`で探せば区別できます。実際のvoiceが単語の先頭ぴったりではない位置を返す場合もあるため、GlyphFlowでは最も近い該当トークンを安全に探します。

## 1文字ずつDOMにしなかった

最初に考えたのは、単語を1文字ずつ`span`へ分割して順番に色を変える方法でした。ただ、この表示で欲しいのは文字ごとの段階的な点灯ではありません。境界が連続的に通過する見た目です。

そこで、単語は1要素のままにし、文字だけを背景グラデーションで切り抜く形にしました。

```html
<span class="kn-word" style="--kn-progress: 42%">caption</span>
```

```css
.kn-word {
color: transparent;
background: linear-gradient(
90deg,
var(--kn-highlight-color, #f3f7ff) 0%,
var(--kn-highlight-color, #8da7ff) var(--kn-progress),
var(--kn-unhighlight-color, #475061) var(--kn-progress),
var(--kn-unhighlight-color, #475061) 100%
);
background-clip: text;
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
```

更新するインラインスタイルは`--kn-progress`だけです。DOMを文字単位に増やさず、単語の途中にも連続した境界を作れます。

## イベントが来ない間をどう見せるか

単語の開始イベントを受け取った直後は、次の単語がいつ始まるか分かりません。それでも、その間ずっと表示を止めると字幕らしく見えません。

GlyphFlowでは、単語の長さ、簡単な文字パターン、rate、直後の句読点から、その単語が終わるまでの時間を軽く見積もります。辞書や音声モデルは使いません。これは実測値ではなく、次のイベントまでの見た目を動かすための仮の時間です。

予測どおりの時刻に100%まで到達させると、次のイベントが少し遅れたときに、単語が完了したまま待ってしまいます。そこで、予測時間では94%まで進め、その後は最大98.5%までゆっくり近づけます。

次の単語境界が届いた瞬間に、直前の単語を100%にします。同時に、2つの開始イベントの`elapsedTime`の差から、直前の単語に実際にかかった時間を得られます。

```ts
const actualDuration = nextBoundary.elapsedTime - currentBoundary.elapsedTime;
```

この値は同じ読み上げセッション内だけで使います。計測値の一部を後続の予測に反映しますが、極端に短い・長い値は学習に使いません。長い文章ほど材料が増える一方、これはvoiceの発音を完全に理解する仕組みではありません。

## イベントをそのまま信じない

ブラウザイベントの扱いでは、見た目より状態管理の方が難しい部分でした。

- 同じ`boundary`が重複して届く
- 順番が逆のイベントが届く
- pause中も`requestAnimationFrame`だけは進んでしまう
- cancel後に古いutteranceのイベントが届く
- 終了まで単語境界を返さないvoiceがある

このため、再生ごとにセッションIDを持ち、現在のセッションと一致しないイベントは無視します。pause中はワイプを進めず、resume後はpause時間を補正します。終了まで有効な単語境界が来なかった場合は、同期できなかったことを`unsupported`として利用側に伝えます。

ここは「すべてのvoiceを支援する」と言い切らないための設計でもあります。音声品質やイベントの粒度は、ライブラリ側で改善できるものではありません。

## 使う側のコードは小さくした

ライブラリの利用側は、テキスト、描画先、voice、rateを渡して`speak()`するだけです。

```ts
import { createKaraokeNarrator } from "@enumura/glyphflow";
import "@enumura/glyphflow/styles.css";

const narrator = createKaraokeNarrator({
text: "Every letter follows the voice.",
target: document.querySelector("#caption")!,
rate: 1,
});

narrator.speak();
```

読み上げに伴う単語ごとの予測値・実測値は`onWordTiming`と`getDiagnostics()`から取得できます。ただし、最後の単語には次の境界がないため、実測時間サンプルはありません。この点も、表示用の推定と、観測できた時間データを混同しないために明示しています。

## まとめ

Web Speech APIの単語境界だけでは、文字ごとの発音時刻は分かりません。それでも、単語を1要素のまま残し、CSSグラデーションの境界を動かし、次のイベントまでの表示を控えめに予測すれば、単語の内側を進む字幕UIは作れます。

実装とデモは以下です。

- GitHub: https://github.com/speech-caption/glyphflow
- npm: https://www.npmjs.com/package/@enumura/glyphflow
- Demo: https://speech-caption.github.io/glyphflow/
Loading