From a697759111f31e116c548b200a26203979aec4d1 Mon Sep 17 00:00:00 2001 From: ronalduQualabs Date: Mon, 1 Jun 2026 15:57:25 -0300 Subject: [PATCH 1/3] docs: multi-audio-track API design --- internal/design/audio-track-api.md | 50 ++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 internal/design/audio-track-api.md diff --git a/internal/design/audio-track-api.md b/internal/design/audio-track-api.md new file mode 100644 index 000000000..74434f192 --- /dev/null +++ b/internal/design/audio-track-api.md @@ -0,0 +1,50 @@ +--- +status: draft +date: 2026-06-01 +--- + +# Multi-Audio-Track API + +A consumer-facing API for discovering and switching audio tracks in Video.js 10, surfacing what SPF already knows internally through the same feature/predicate/selector pattern used by `streamType` and `textTrack`. + +## Problem + +SPF already parses audio renditions from HLS multivariant playlists and manages `selectedAudioTrackId` internally. The segment loading, buffer management, and track resolution behaviors are all in place. None of this is surfaced to the outside world in a stable, typed way. Building a language-switcher UI today means reaching directly into `media.engine.state` — an internal SPF primitive with no stability contract. + +## Why the Text Track Pattern Doesn't Transfer Directly + +The `textTrack` feature reads from the native browser `textTracks` API on the media element. This works because SPF's `syncTextTracks` behavior creates real `` DOM elements, which the browser automatically registers in `mediaElement.textTracks`. When SPF adds a track, the browser fires a native `addtrack` event that the core feature listens to. + +Audio tracks work differently. SPF manages audio rendition switching entirely at the MSE byte-stream level — it selects which HLS rendition's segments feed the audio `SourceBuffer`. There is no ``-element equivalent for audio. The native `mediaElement.audioTracks` list remains empty even during multi-language HLS playback. Reading from it yields nothing. + +The Mux media-tracks polyfill extends `HTMLMediaElement` with a conformant `audioTracks` property, but unlike text tracks where the browser populates the list from `` elements SPF creates, audio tracks would require actively pushing SPF's internal state into the polyfill on every change. That is a synchronization layer with a new external dependency and no reduction in complexity. It is a separate, independent decision from surfacing the API. + +The right pattern for audio tracks is therefore: the adapter exposes a custom property and event, the core feature reads from that, the store surfaces it through a selector. This is the same approach used by `streamType`, which also has no native DOM equivalent. + +## Proposed Architecture + +**Adapter (`SimpleHlsMedia`, in core).** `SimpleHlsMedia` is the boundary where SPF internals meet the rest of the stack. It gains an `audioTracks` read-only property (returning a list of track objects with `id`, `label`, `language`, `kind`, and `enabled`), a `selectAudioTrack(id)` method, and fires an `audiotrackschange` event whenever the track list or active selection changes. + +Internally it uses SPF's `effect()` to watch `presentation` and `selectedAudioTrackId` signals and dispatch the event when either changes. The effect is created fresh on each `src` assignment and torn down on `destroy()`, matching the engine lifecycle. + +The track shape maps SPF's `AudioTrack` fields to a clean public surface: `name` becomes `label`, `language` carries over, and `enabled` is derived by comparing each track's `id` to `selectedAudioTrackId`. SPF's `AudioTrack` has no `kind` field — the property defaults to `'main'`, appropriate for HLS audio renditions today. + +**Core feature (`audioTrackFeature`).** A new feature in `packages/core/src/dom/store/features/` follows the exact structure of `textTrackFeature`. A predicate (`isMediaAudioTrackCapable`) checks whether the media object exposes `audioTracks` and `selectAudioTrack`, using the same shape as `isMediaStreamTypeCapable`. If capable, the feature syncs on `audiotrackschange` and stores the result. The state type (`MediaAudioTrackState`) and capability interface (`MediaAudioTrackCapability`) live next to their text track equivalents in `core/media/state.ts` and `core/media/types.ts`. + +**Selector.** `selectAudioTrack` is added to the selectors file parallel to `selectTextTrack`. UI components subscribe to track list changes and invoke switching through this selector. + +**Feature presets.** `audioTrackFeature` is added to all four presets — video, audio, live-video, live-audio. Multi-language audio applies to every stream type, including audio-only. + +## What This Does Not Change + +SPF internals are untouched. The `selectAudioTrack` behavior, `resolveAudioTrack`, `setupAudioBufferActors`, and `loadAudioSegments` all continue working as-is. The adapter reads `engine.state.presentation` and writes `engine.state.selectedAudioTrackId` — both already public signals on the composition. + +Other adapters (`HlsJsMedia`, `NativeHlsMedia`) are not in scope here. The predicate-based design lets them opt in independently whenever they implement the same three-part contract. + +## Related Work + +**multi-language-audio** (`internal/design/spf/features/multi-language-audio.md`) — the SPF-layer feature this API surfaces. The API described here covers its Tier 1 (recognition and exposure) and the start of Tier 2 (programmatic selection). Mid-stream switching correctness remains an SPF concern. + +**engine-adapter-integration** (`internal/design/spf/features/engine-adapter-integration.md`) — the `shareSignals` mechanism the adapter reads and writes through. + +**text-track-architecture** (`internal/design/spf/text-track-architecture.md`) — the reference implementation whose structure `audioTrackFeature` follows at the feature layer, and whose DOM integration path audio tracks deliberately diverge from at the SPF layer. From 72a33b28410fceceaec4ad8755087d2830a8a7f1 Mon Sep 17 00:00:00 2001 From: ronalduQualabs Date: Tue, 2 Jun 2026 16:16:38 -0300 Subject: [PATCH 2/3] docs: align audio-track design with MediaTracksLayer pattern from #1609 --- internal/design/audio-track-api.md | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/internal/design/audio-track-api.md b/internal/design/audio-track-api.md index 74434f192..5d12168a9 100644 --- a/internal/design/audio-track-api.md +++ b/internal/design/audio-track-api.md @@ -11,27 +11,19 @@ A consumer-facing API for discovering and switching audio tracks in Video.js 10, SPF already parses audio renditions from HLS multivariant playlists and manages `selectedAudioTrackId` internally. The segment loading, buffer management, and track resolution behaviors are all in place. None of this is surfaced to the outside world in a stable, typed way. Building a language-switcher UI today means reaching directly into `media.engine.state` — an internal SPF primitive with no stability contract. -## Why the Text Track Pattern Doesn't Transfer Directly +## Architecture -The `textTrack` feature reads from the native browser `textTracks` API on the media element. This works because SPF's `syncTextTracks` behavior creates real `` DOM elements, which the browser automatically registers in `mediaElement.textTracks`. When SPF adds a track, the browser fires a native `addtrack` event that the core feature listens to. +Audio track exposure uses the same `media-tracks` polyfill layer already in the codebase — no custom properties or events are needed on the adapter. PR [#1609](https://github.com/videojs/v10/pull/1609) establishes the pattern for `HlsJsMedia` and `NativeHlsMedia`. -Audio tracks work differently. SPF manages audio rendition switching entirely at the MSE byte-stream level — it selects which HLS rendition's segments feed the audio `SourceBuffer`. There is no ``-element equivalent for audio. The native `mediaElement.audioTracks` list remains empty even during multi-language HLS playback. Reading from it yields nothing. +**Media-tracks layer.** `MediaTracksLayer` (see `packages/core/src/dom/media/media-tracks-layer.ts`) wraps `HTMLMediaElementLayer` with `MediaTracksMixin`, introduced in PR [#1609](https://github.com/videojs/v10/pull/1609). This gives any host that adds the layer a conformant `audioTracks` (and `videoTracks`) property, along with `addAudioTrack()` / `addVideoTrack()` / `addRendition()` helpers and standard `change` events — the same shape the browser provides natively on `mediaElement.audioTracks`. -The Mux media-tracks polyfill extends `HTMLMediaElement` with a conformant `audioTracks` property, but unlike text tracks where the browser populates the list from `` elements SPF creates, audio tracks would require actively pushing SPF's internal state into the polyfill on every change. That is a synchronization layer with a new external dependency and no reduction in complexity. It is a separate, independent decision from surfacing the API. +**Adapter (`SimpleHlsMedia`).** Call `addLayer(this, new MediaTracksLayer())` in the constructor, matching what `NativeHlsMedia` already does. No custom property or event is needed — the layer owns `audioTracks`. -The right pattern for audio tracks is therefore: the adapter exposes a custom property and event, the core feature reads from that, the store surfaces it through a selector. This is the same approach used by `streamType`, which also has no native DOM equivalent. +**Extension (`SimpleHlsMediaTracks`).** A `MediaExtension` (following `HlsJsMediaTracks` in `packages/core/src/dom/media/hls-js/media-tracks.ts`) installs onto `SimpleHlsMedia`. It uses SPF's `effect()` to watch `presentation` and `selectedAudioTrackId` signals, calls `removeAudioTracks(media)` + `media.addAudioTrack(kind, name, lang)` when the track list changes, and sets `audioTrack.enabled` to reflect the active selection. When `media.audioTracks` fires `change` (user or programmatic), the extension writes the new selection back to `engine.state.selectedAudioTrackId`. -## Proposed Architecture +**Core feature (`audioTrackFeature`).** A new feature in `packages/core/src/dom/store/features/` follows the exact structure of `textTrackFeature`. The predicate (`isMediaAudioTrackCapable`) checks whether the media object exposes `audioTracks` — using the same shape as `isMediaStreamTypeCapable`. Because `MediaTracksLayer` provides the standard `AudioTrackList` (including `addtrack`, `removetrack`, and `change` events), the feature can sync via `audioTracks.onchange` the same way `textTrackFeature` syncs via `textTracks`. The state type (`MediaAudioTrackState`) and capability interface (`MediaAudioTrackCapability`) live next to their text track equivalents. -**Adapter (`SimpleHlsMedia`, in core).** `SimpleHlsMedia` is the boundary where SPF internals meet the rest of the stack. It gains an `audioTracks` read-only property (returning a list of track objects with `id`, `label`, `language`, `kind`, and `enabled`), a `selectAudioTrack(id)` method, and fires an `audiotrackschange` event whenever the track list or active selection changes. - -Internally it uses SPF's `effect()` to watch `presentation` and `selectedAudioTrackId` signals and dispatch the event when either changes. The effect is created fresh on each `src` assignment and torn down on `destroy()`, matching the engine lifecycle. - -The track shape maps SPF's `AudioTrack` fields to a clean public surface: `name` becomes `label`, `language` carries over, and `enabled` is derived by comparing each track's `id` to `selectedAudioTrackId`. SPF's `AudioTrack` has no `kind` field — the property defaults to `'main'`, appropriate for HLS audio renditions today. - -**Core feature (`audioTrackFeature`).** A new feature in `packages/core/src/dom/store/features/` follows the exact structure of `textTrackFeature`. A predicate (`isMediaAudioTrackCapable`) checks whether the media object exposes `audioTracks` and `selectAudioTrack`, using the same shape as `isMediaStreamTypeCapable`. If capable, the feature syncs on `audiotrackschange` and stores the result. The state type (`MediaAudioTrackState`) and capability interface (`MediaAudioTrackCapability`) live next to their text track equivalents in `core/media/state.ts` and `core/media/types.ts`. - -**Selector.** `selectAudioTrack` is added to the selectors file parallel to `selectTextTrack`. UI components subscribe to track list changes and invoke switching through this selector. +**Selector.** `selectAudioTrack` is added to the selectors file parallel to `selectTextTrack`. UI components subscribe to track list changes; switching is done by setting `audioTrack.enabled = true` on the desired track. **Feature presets.** `audioTrackFeature` is added to all four presets — video, audio, live-video, live-audio. Multi-language audio applies to every stream type, including audio-only. From d7dfa4f96a37b61eccd22cada1cb5b1ebda06439 Mon Sep 17 00:00:00 2001 From: ronalduQualabs Date: Tue, 16 Jun 2026 11:19:46 -0300 Subject: [PATCH 3/3] docs(core): update audio-track-api design to reflect #1664 and #1605 --- internal/design/audio-track-api.md | 83 ++++++++++++++++++++++++++---- 1 file changed, 72 insertions(+), 11 deletions(-) diff --git a/internal/design/audio-track-api.md b/internal/design/audio-track-api.md index 5d12168a9..4560148b1 100644 --- a/internal/design/audio-track-api.md +++ b/internal/design/audio-track-api.md @@ -1,6 +1,7 @@ --- status: draft date: 2026-06-01 +updated: 2026-06-16 --- # Multi-Audio-Track API @@ -11,27 +12,85 @@ A consumer-facing API for discovering and switching audio tracks in Video.js 10, SPF already parses audio renditions from HLS multivariant playlists and manages `selectedAudioTrackId` internally. The segment loading, buffer management, and track resolution behaviors are all in place. None of this is surfaced to the outside world in a stable, typed way. Building a language-switcher UI today means reaching directly into `media.engine.state` — an internal SPF primitive with no stability contract. -## Architecture +## What Is Already Done + +- **#1605** — SPF internal switching: `selectedAudioTrackId`, language-aware default selection, mid-stream switches. +- **#1664** — `MediaTracksMixin` (the `audioTracks` DOM infrastructure) + `HlsJsMediaMediaTracksMixin` wired to `HlsMedia` (hls.js path). Types and capability interfaces in `packages/core/src/core/media/types.ts`. + +`SimpleHlsMedia` (the SPF path) is not wired. No store feature or selector exists yet. + +## Remaining Work + +Three things are still missing: + +### 1. Wire `SimpleHlsMedia` (the SPF adapter) + +`SimpleHlsMedia` (`packages/core/src/dom/media/simple-hls/index.ts`) is currently: + +```ts +export class SimpleHlsMedia extends SimpleHlsMediaMixin(HTMLVideoElementHost) {} +``` + +It needs `MediaTracksMixin` applied and a `SimpleHlsMediaTracks` extension that bridges SPF signals to the track lists. + +**The extension** watches two SPF signals via `effect()`: -Audio track exposure uses the same `media-tracks` polyfill layer already in the codebase — no custom properties or events are needed on the adapter. PR [#1609](https://github.com/videojs/v10/pull/1609) establishes the pattern for `HlsJsMedia` and `NativeHlsMedia`. +- `presentation` — when a new manifest loads, clear and re-add `AudioTrack` objects from `engine.state.audioTracks` +- `selectedAudioTrackId` — set `audioTrack.enabled = true` on the matching track -**Media-tracks layer.** `MediaTracksLayer` (see `packages/core/src/dom/media/media-tracks-layer.ts`) wraps `HTMLMediaElementLayer` with `MediaTracksMixin`, introduced in PR [#1609](https://github.com/videojs/v10/pull/1609). This gives any host that adds the layer a conformant `audioTracks` (and `videoTracks`) property, along with `addAudioTrack()` / `addVideoTrack()` / `addRendition()` helpers and standard `change` events — the same shape the browser provides natively on `mediaElement.audioTracks`. +In the other direction: when `media.audioTracks` fires `change`, write the selected track's id back to `engine.state.selectedAudioTrackId`. -**Adapter (`SimpleHlsMedia`).** Call `addLayer(this, new MediaTracksLayer())` in the constructor, matching what `NativeHlsMedia` already does. No custom property or event is needed — the layer owns `audioTracks`. +The template to follow is `HlsJsMediaMediaTracksMixin` in `packages/core/src/dom/media/hls/media-tracks.ts` — same structure, different engine event API (SPF `effect()` instead of hls.js events). -**Extension (`SimpleHlsMediaTracks`).** A `MediaExtension` (following `HlsJsMediaTracks` in `packages/core/src/dom/media/hls-js/media-tracks.ts`) installs onto `SimpleHlsMedia`. It uses SPF's `effect()` to watch `presentation` and `selectedAudioTrackId` signals, calls `removeAudioTracks(media)` + `media.addAudioTrack(kind, name, lang)` when the track list changes, and sets `audioTrack.enabled` to reflect the active selection. When `media.audioTracks` fires `change` (user or programmatic), the extension writes the new selection back to `engine.state.selectedAudioTrackId`. +### 2. Add `audioTrackFeature` to the store -**Core feature (`audioTrackFeature`).** A new feature in `packages/core/src/dom/store/features/` follows the exact structure of `textTrackFeature`. The predicate (`isMediaAudioTrackCapable`) checks whether the media object exposes `audioTracks` — using the same shape as `isMediaStreamTypeCapable`. Because `MediaTracksLayer` provides the standard `AudioTrackList` (including `addtrack`, `removetrack`, and `change` events), the feature can sync via `audioTracks.onchange` the same way `textTrackFeature` syncs via `textTracks`. The state type (`MediaAudioTrackState`) and capability interface (`MediaAudioTrackCapability`) live next to their text track equivalents. +A new feature in `packages/core/src/dom/store/features/audio-track.ts` following the exact structure of `textTrackFeature` (`text-track.ts`). It: -**Selector.** `selectAudioTrack` is added to the selectors file parallel to `selectTextTrack`. UI components subscribe to track list changes; switching is done by setting `audioTrack.enabled = true` on the desired track. +- Defines `MediaAudioTrackState` — the list of available audio tracks and which is currently enabled +- Uses the predicate `isMediaAudioTrackCapable` (checks `'audioTracks' in media`) — same shape as `isMediaStreamTypeCapable` +- Syncs `media.audioTracks` into store state via `addtrack`, `removetrack`, and `change` listeners +- Re-syncs on `loadstart` (new source clears tracks) -**Feature presets.** `audioTrackFeature` is added to all four presets — video, audio, live-video, live-audio. Multi-language audio applies to every stream type, including audio-only. +**`selectAudioTrack`** is added to `packages/core/src/dom/store/selectors.ts` parallel to `selectTextTrack`. + +### 3. Register `audioTrackFeature` in all four presets + +`audioTrackFeature` is added to `videoFeatures`, `audioFeatures`, `liveVideoFeatures`, and `liveAudioFeatures` in `packages/core/src/dom/store/features/presets.ts`. Multi-language audio applies to every stream type including audio-only streams. + +## Architecture + +The full stack, bottom to top: + +```text +┌──────────────────────────────────────────────────┐ +│ selectAudioTrack (selectors.ts) │ ← UI reads from here +├──────────────────────────────────────────────────┤ +│ audioTrackFeature (store/features/) │ ← syncs DOM audioTracks → store +├──────────────────────────────────────────────────┤ +│ SimpleHlsMediaTracks extension │ ← bridges SPF signals ↔ DOM tracks +├──────────────────────────────────────────────────┤ +│ MediaTracksMixin on SimpleHlsMedia │ ← gives adapter audioTracks property +└──────────────────────────────────────────────────┘ + ▲ SPF engine (unchanged — #1605 landed switching) +``` + +`HlsMedia` (hls.js path) already has Layers 1 and 2 via #1664. `SimpleHlsMedia` (SPF path) needs the same. ## What This Does Not Change -SPF internals are untouched. The `selectAudioTrack` behavior, `resolveAudioTrack`, `setupAudioBufferActors`, and `loadAudioSegments` all continue working as-is. The adapter reads `engine.state.presentation` and writes `engine.state.selectedAudioTrackId` — both already public signals on the composition. +SPF internals are untouched. The `selectAudioTrack` behavior, `resolveAudioTrack`, `setupAudioBufferActors`, and `loadAudioSegments` all continue working as-is. The extension reads `engine.state.presentation` and writes `engine.state.selectedAudioTrackId` — both already public signals on the composition. + +`HlsMedia` and `NativeHlsMedia` are not in scope. The predicate-based design lets them opt in independently — `HlsMedia` is already in via #1664. -Other adapters (`HlsJsMedia`, `NativeHlsMedia`) are not in scope here. The predicate-based design lets them opt in independently whenever they implement the same three-part contract. +## Files to Create / Modify + +| File | Change | +| --- | --- | +| `packages/core/src/dom/media/simple-hls/audio-tracks.ts` | New — `SimpleHlsMediaTracks` extension (Layer 2) | +| `packages/core/src/dom/media/simple-hls/index.ts` | Apply `MediaTracksMixin`, wire extension | +| `packages/core/src/dom/store/features/audio-track.ts` | New — `audioTrackFeature` + `MediaAudioTrackState` (Layer 3) | +| `packages/core/src/dom/store/selectors.ts` | Add `selectAudioTrack` (Layer 4) | +| `packages/core/src/dom/store/features/presets.ts` | Register `audioTrackFeature` in all four presets | ## Related Work @@ -39,4 +98,6 @@ Other adapters (`HlsJsMedia`, `NativeHlsMedia`) are not in scope here. The predi **engine-adapter-integration** (`internal/design/spf/features/engine-adapter-integration.md`) — the `shareSignals` mechanism the adapter reads and writes through. -**text-track-architecture** (`internal/design/spf/text-track-architecture.md`) — the reference implementation whose structure `audioTrackFeature` follows at the feature layer, and whose DOM integration path audio tracks deliberately diverge from at the SPF layer. +**text-track-architecture** (`internal/design/spf/text-track-architecture.md`) — the reference implementation whose structure `audioTrackFeature` follows at the feature/store layer. + +**PR #1664** — landed `MediaTracksMixin` infrastructure and `HlsJsMediaMediaTracksMixin`. The canonical reference for the mixin pattern this design extends to `SimpleHlsMedia`.