1871143 - Feature request: Support WebVTT-based synthesized audio description in video

Status: UNCONFIRMED

(Core :: Audio/Video, enhancement)

Description

[Adrian Roselli]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0

Steps to reproduce:

This is partly bug report and partly feature request.

I attempted to have the browser generate synthesized audio descriptions for a video using WebVTT per WHATWG HTML guidance.

Sample:
https://adrianroselli.com/2023/12/ad-support-in-html-video.html#Synthesized

Code:

<video preload="metadata" controls poster="star-video_poster.jpg">
    <source src="star-video.mp4" type="video/mp4">
    <track label="English" kind="subtitles" srclang="en-us" src="star-video_base.vtt" default>
    <track label="Audio Description" kind="descriptions" srclang="en-us" src="star-video_AD.vtt">
    Sorry, your browser doesn’t support embedded videos, but don’t worry, you can <a href="star-video.mp4">download it</a>. The <a href="star-video_base.vtt">caption file</a> is also available in case your video player can import it.
</video>

Actual results:

Nothing. No acknowledgment of any additional track.

Expected results:

I expected to hear a synthesized voice speaking the audio description in time with the video based on the WebVTT file.

Current / updated WCAG advisory technique:
https://www.w3.org/WAI/WCAG22/Techniques/html/H96

Relevant parsing rules from WHATWG HTML:
https://html.spec.whatwg.org/multipage/media.html#playing-the-media-resource

Latest update from WebVTT CG Report:

The majority of the current version of this specification is dedicated to describing how to use WebVTT files for captioning or subtitling. There is minimal information about chapters and time-aligned metadata and nothing about video descriptions at this stage.
https://w3c.github.io/webvtt/#introduction

In other words, I would love to see this supported but if WHATWG HTML is punting to WebVTT and WebVTT has not fully specified it, then perhaps Firefox can either:

1. take the lead and implement this, helping cement behaviors; or
2. push for clarity in WHATWG HTML using its position of influence.

Related…
meta tracking bug for WebVTT:
https://bugzilla.mozilla.org/show_bug.cgi?id=webvtt

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.