Proposal: Native Word & Syllable Media Synchronization (Timed Media Streams)

[hemanth]

Proposal: Native Word & Syllable Media Synchronization (Timed Media Streams)

Introduction

Web applications increasingly require sub-word, syllable-level, and continuous time-synchronized text synchronization with both audio and video media streams. Key use cases include:

• Real-time music karaoke and chanting teleprompters
• Video closed-captioning with word tracking for cognitive accessibility
• Educational video lectures with synchronized transcript tracking
• Language pronunciation learning, speech therapy, and dubbing (ADR)

Today, the web platform lacks a native, declarative primitive for progressive, sub-word media alignment across HTMLMediaElement (<audio> and <video>).

Developers currently resort to 60fps JavaScript requestAnimationFrame loops querying stepped, jittery HTMLMediaElement.currentTime values (which update at 4Hz–15Hz). This causes clock drift, frame drops, battery drain, and main-thread layout thrashing. Furthermore, existing standards like WebVTT and TTML fail on agglutinative, inflected, and compounding languages (such as Sanskrit, German, Finnish, and Arabic) where acoustic tokens diverge from semantic words due to morphological sandhi splits.

Read the Complete Explainer Gist

---

Key Proposals

1. CSS Media Timeline Bindings & Pseudo-Elements

Allow CSS animations and progressive fill sweeps to bind directly to media playback hardware clocks without JavaScript RAF polling:

/* Style words dynamically based on media playback status */
::cue-word(:active) {
  color: var(--gold-primary);
  /* Hardware-accelerated sweep driven by media clock */
  background: linear-gradient(
    to right,
    var(--gold-active) 0%,
    var(--gold-active) var(--cue-word-progress),
    var(--text-color) var(--cue-word-progress),
    var(--text-color) 100%
  );
  -webkit-background-clip: text;
}

2. Extended Timed Text Model (<track kind="lyrics">)

Add first-class support for sub-word morphemes and morphological sandhi splits in <track> elements for <audio> and <video>:

<audio id="audio-player" src="audio.mp3">
  <track kind="lyrics" src="lyrics.vtt" srclang="sa" default>
</audio>

<video id="video-player" src="video.mp4" controls>
  <track kind="lyrics" src="video-lyrics.vtt" srclang="en" default>
</video>

3. MediaAnimationTimeline Interface

A Web Animations API (WAAPI) timeline bound directly to the audio DAC / video Presentation Timestamps (PTS):

const audioElement = document.querySelector('audio');
const timeline = new MediaAnimationTimeline(audioElement);

wordElement.animate(
  [{ backgroundPosition: '0% 0%' }, { backgroundPosition: '100% 0%' }],
  { timeline: timeline, timeRange: { start: 10000, end: 10820 }, fill: 'both' }
);

---

Feedback

We welcome feedback, discussion, and input from browser implementers, media streaming engineers, and accessibility experts on this thread.

• Full Explainer: https://gist.github.com/hemanth/af388c6a57238f893bcb56194b8f7566