Bugzilla

Reporter

Comment 2

•

4 years ago

https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Audio_codecs
I can somehow understand that you don't allow to play H264 (supported) + inefficient AC3 (patent expired February 1, 2017) in mkv, although it's super sad for existing files (you know what I mean; bug 1210727), so I wanted to convert to modern codecs, but still use mkv/webm. It would be nice if FLAC (like Opus by Xiph.org) would optionally also work with the open-standard container format.

Comment 3

•

4 years ago

(In reply to Jean-Yves Avenard [:jya] from comment #1)

:kinetik is this something we want to support?

Given that it already works in Chrome (via Matroska) and we already support FLAC in MP4, adding support to Firefox seems reasonable.

Add support to libnestegg is trivial, I've done that on a branch here: https://github.com/kinetiknz/nestegg/commit/3b6054d508600b034cacfd496a0358c165bc09bb

We'll also need to plumb this inside Firefox by handling the new NESTEGG_CODEC_FLAC track type in WebMDemuxer and anywhere else modifications are needed.

It's probably worth testing whether Chrome accepts files with doctype="webm" containing FLAC as if they're Matroska files or rejects them to preserve the stricter WebM format.

Flags: needinfo?(kinetik)

Reporter

Comment 4

•

4 years ago

Thank you! :)
If Chrome played some FLAC webm files and some not, it would be good if Chrome and ffmpeg could create a similar small patch.
Lossless codecs should really be converted to FLAC.

Comment 5

•

4 years ago

Attached file Bug 1606822 - Import libnestegg from upstream for FLAC support. (obsolete) — Details

Comment 6

•

4 years ago

Attached file Bug 1606822 - Allow WebMDecoder to handle FLAC tracks. (obsolete) — Details

Depends on D59072

Comment 7

•

4 years ago

Attached patches are sufficient to play flac.webm. Still need tests in libnestegg and Firefox before this is ready for review.

Comment 8

•

4 years ago

(In reply to Matthew Gregan [:kinetik] from comment #3)

It's probably worth testing whether Chrome accepts files with doctype="webm" containing FLAC as if they're Matroska files or rejects them to preserve the stricter WebM format.

Chrome plays the file irrespective of doctype. I updated flac.webm's doctype from "matroska" to "webm" to verify this.

Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail

Updated

•

4 years ago

Assignee: nobody → kinetik

Status: NEW → ASSIGNED

Comment 9

•

4 years ago

Adding priority to get out of triage queue. Please adjust as you see fit.

Priority: -- → P2

Reporter

Comment 10

•

4 years ago

(For the proper webm doctype it seems ffmpeg only needs a FLAC entry here and it should be mentioned here.)

Jean-Yves Avenard [:jya]

Comment 11

•

4 years ago

(In reply to Matthew Gregan [:kinetik] from comment #3)

Given that it already works in Chrome (via Matroska) and we already support FLAC in MP4, adding support to Firefox seems reasonable.

I'm afraid with that logic, we would support any codecs in webm. Because AFAIK, Chrome handles mkv file exactly the same as webm. So all codecs are allowed.

Reporter

Updated

•

4 years ago

Keywords: parity-edge

OS: Linux → All

Hardware: x86_64 → All

Comment 12

•

4 years ago

•

Edited

(In reply to Jean-Yves Avenard [:jya] from comment #11)

I'm afraid with that logic, we would support any codecs in webm. Because AFAIK, Chrome handles mkv file exactly the same as webm. So all codecs are allowed.

Yes, that seems to be the path we are on ever since we started accepting "matroska" doctype files in bug 1377278. Comment 7 in that bug covers my thoughts back then. Chrome's behaviour leaves us in an awkward position. If you think supporting FLAC in WebM/Matroska is a mistake, we can leave the codec support support as-is and WONTFIX this bug (and possibly bug 1422891?).

Updated

•

4 years ago

Type: defect → enhancement

Priority: P2 → P3

Updated

•

4 years ago

Summary: Firefox does not play FLAC in webm → Firefox does not play FLAC in Matroska

Reporter

Comment 13

•

4 years ago

I think the difference to bug 1422891 is that webm only wanted to allow free codecs and that is a honorable value on its own.

Comment 14

•

4 years ago

Chromium supports PCM and H264/AVC1 in Matroska container.

Therefore any semblance of a rule to restrict WebM to Vorbis, Opus, VP* are moot as to Chrome, Chromium (under *oogle umbrella) being in accord with any such non-binding agreement.

When saved as a WebM file the media is playable at Chromium - as long as the file is not directly navigated to https://bugs.chromium.org/p/chromium/issues/detail?id=999580.

Browser implementers' operate on their own schedule, can and do remove any API they want to at any time.

All of these audio codec encoders and decoders can be compiled into a front-end controlled repository or ingle script, in order to avoid having to ask browser implementers for anything, e.g., (https://github.com/guest271314/native-messaging-espeak-ng/blob/master/host/install_host.sh#L40) when decided to compose code to parse SSML and output audio, included Opus encoding and decoding in the build script, without waiting another N years for browser authors to implement SSML parsing or delivering the output of local speech synthesis engine as Opus encoded file.

Another option would be to abandon HTML <video> element and corresponding decoders altogether and port mpv to JavaScript. Or, use the local binary with Native Messaging, e.g., when Chromium was not resizing video used mkvmerge to achieve that requirement (https://github.com/guest271314/native-messaging-mkvmerge).

Comment 15

•

4 years ago

Technically all that is required is to create a generic decodeAudioData() function that accepts an ArrayBuffer then feed the Float32Array to AudioWorklet, where implemented, or use createScriptProcessor(). opus-tools and mkvmerge (very well maintained) should provide the means to achieve the requirement. If necessary the algorithm could include conversion to WAV (https://github.com/mohayonao/wav-decoder/blob/master/index.js) in between or as output

Audio (<any>) as ArrayBuffer => WAV => Float32Array => set channel data without output stream at AudioWorklet

e.g., https://github.com/guest271314/native-messaging-espeak-ng/blob/bash-audioworklet/app/audioWorklet.js#L32 where technically the input can be an infinite input stream or single-use input retrieved any number of ways, from fetch() to MesagePort.postMessage() to RTCDataChannel(), et al. within a Worker or AudioWorkletGlobalScope or from main thread.

Perhaps what is required is to build the universal media decoer from scratch then post the deliverable once completed by the users that will use (and maintain) the code? If there is resistance to including various encoders and decoders into base source code, will those same individuals and interests that resisted, once included, be the maintainers of that code?

Comment 16

•

4 years ago

Alternatively, a canonical means to programmatically create a "fake" device could be used which can be selected by getUserMedia() to playback the media. Firefox has not yet implemented Native File System, though equivalent is possible using File API to create a local script that can be executed and output accessible to browser, either by streaming or piping directly to a sink, or by getting the file.

An experiment to use Chromium as both a utility and a browser, where inotify-tools is used to montitor file attribute changes. Any shell script can be written in the browser, saved in the local filesystem, then executed, where STDOUT is capable of being retrieved, without using Native File System or WebAssembly, meaning the ability to execute arbitrary shell scripts on the local OS from browser input

<!DOCTYPE html>
<html>

<head>
<meta charset="utf-8"/>
  <title>Choose directory, get writable permission, write without executing close()</title>
</head>

<body>
  <p id="choose_directory_get_writable_permission">Choose directory, get writeable permission, execute script</p>
  <script>
    let input = `<voice name="Storm">Test</voice>`, dir, status, execFileHandle, inputFileHandle, outputFileHandle, writer;
    const ac = new AudioContext();

    document.getElementById("choose_directory_get_writable_permission").onclick = async e => {
      try {
        if (!(dir instanceof FileSystemDirectoryHandle && dir.name === "localscripts") && status !== "granted") {
          dir = await self.chooseFileSystemEntries({type:"openDirectory"});
          status = await dir.requestPermission({writable:true});
        }
        inputFileHandle = await dir.getFile("input.txt", {create:true});
        execFileHandle = await dir.getFile("run.txt", {create: true});   
        writer = await inputFileHandle.createWriter({keepExistingData: false});
        await writer.write(0, new Blob([input], {
          type: "text/plain"
        }));
        await writer.close();
        writer = await execFileHandle.createWriter({keepExistingData: false});
        await writer.write(0, new Blob([], {
          type: "text/plain"
        }));
        // wait 25 for file to be written
        await Promise.all([writer.close(), new Promise(resolve => setTimeout(resolve, 25))]);
        outputFileHandle = await dir.getFile("output.wav", {create: false});
        const buffer = await (await outputFileHandle.getFile()).arrayBuffer();
        await Promise.all(["input.txt", "output.wav"].map(fn => dir.removeEntry(fn)));
        const ab = await ac.decodeAudioData(buffer);
        const source = ac.createBufferSource();
        source.buffer = ab;
        source.onended = e => {
          source.disconnect();
          console.log(e);
        }
        source.connect(ac.destination);
        source.start();
      } catch (e) {
          console.error(e);
      }
    }
  </script>
</body>

</html>

#!/bin/bash
# script.sh
/usr/bin/espeak-ng -m -f $HOME/localscripts/input.txt -w $HOME/localscripts/output.wav

``
#!/bin/bash

notify_directory_modifications.sh

requestLocalScripts() {
while true; do
inotifywait -e attrib -e close $HOME/localscripts/run.txt | while read a b;
do
$HOME/localscripts/script.sh
echo "$$ $a $b"
done
done
}

requestLocalScripts

chromium-browser --utility-and-browser --utility-cmd-prefix="/path/to/script.sh"


Still has issues, being not yet capable of running as utility and browser, though am able to get expected output at `file:` protocol, given have not found any documentation of usage of the flags besides debugging, am probably missing or including a small element that needs adjustment.

Would suggest exploring using Native Messaging to achieve results not available by default in the browser, in lieu of using `WebAssembly`, which requires compiling a binary instead of using the binary installed on the machine. Not for the messaging protocol itself, which has limitations, but for the ability to execute arbitrary shell scripts and get the output at the main thread.

Reporter

Comment 17

•

4 years ago

If using FLAC with webm (or mkv) would be denied then we basically say: "Please do not use the free container format for AV1+FLAC, please use mp4 instead".
MP4 could always be used, however, webm is free and popular. Until now we had a mental barrier between a not-so-free container format with all codecs (mp4) and the free container format restricted to free codecs (webm). Chromium broke that taboo (by accident?) and maybe we should file a bug asking them to restore the restriction of webm to free codecs. They could still support mkv for everything, but at least we would get some guarantees back that webm files must consist of free codecs.
Offtopic (bug 1422891): Would allowing usage of mkv with the same set of codecs that are supported with mp4 be completely wrong? People could decide to prefer the mkv doctype over webm even for free codecs, but that's already supported. Non-free codecs are also already supported via mp4. The concern I see is that people could then completely switch over to the mkv file type and ignore our wish to prefer webm. I did not really ask for this as it needs its own consideration.
Re guest271314: Personally I'm not a friend of adding more flexibility and complexity - which would also delay this further. To replace media players like VLC, couldn't any unsupported audio codec be transcoded to opus or flac using wasm (preferably written in Rust)? Filing a different bug for your suggestion could be better.

https://bugs.chromium.org/p/chromium/issues/detail?id=980822

Comment 18

•

4 years ago

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #17)

MP4 could always be used, however, webm is free and popular. Until now we had a mental barrier between a not-so-free container format with all codecs (mp4) and the free container format restricted to free codecs (webm). Chromium broke that taboo (by accident?) and maybe we should file a bug asking them to restore the restriction of webm to free codecs.

AFAICT use of H264 and AVC1 (and openH264) is "free" for non-commercial usage per memos published by the "owners"?

They could still support mkv for everything, but at least we would get some guarantees back that webm files must consist of free codecs.

Playback of the .mkv file is not straightforward at Chromium.

Re guest271314: Personally I'm not a friend of adding more flexibility and complexity - which would also delay this further. To replace media players like VLC, couldn't any unsupported audio codec be transcoded to opus or flac using wasm (preferably written in Rust)? Filing a different bug for your suggestion could be better.

Yes. Using Rust WASM is possible. Downloaded Rust for wasm-bindgen. However, for own experiments using Native Messaging was simpler for legacy, essentially obsolete 32-bit machine. The .cargo file filled up just to execute espeak-ng in Web environment. Executing the script directly on the local OS achieved the same result without installing Rust and creating WASM to reflecct what is already installed on the local OS.

Just sharing options that have explored to workaround what is not currently possible by default in web platform domain.

Browser vendors appear to have concerns about "free", "non-free". At the front end that might not be the primary concern. Some might simply want to test (https://gist.githubusercontent.com/guest271314/1fcb5d90799c48fdf34c68dd7f74a304/raw/fa9edb61431b5bc7d7432335ca3ece5763f6209b/UseItBreakItFileBugsRequestFeaturesTestTestTestTestToThePointItBreaks.txt, https://next.plnkr.co/plunk/021iNc) the technology, develop various technologies further, or use the codecs and containers for other interests potentially outside of the use cases implementers have considered.

Not sure what bug to file? The bugs have been filed. Users in the field are asking, over and over again, to support at least decoding Matroska containers having possibly, arbitrary, in this case specific codecs for the track. The technology exists to decode Flac to a representation playable on the Web. Still users ask FOSS browser implementers more to standardize decoding (and encoding) of various media containers and codecs. WebCodecs might offer some means to pipe various codecs to the front-end. Eventually users in the field might explore ways to use whatever containers and codecs they decide to outside of the purview of specification authors or implementers, yet still using their browser to do so.

Comment 19

•

4 years ago

Will Web Audio decodeAudioData() be able to handle the FLAC file at Firefox after changes are made to the source code to fix this bug?

Updated

•

4 years ago

Assignee: kinetik → nobody

Status: ASSIGNED → NEW

Reporter

Updated

•

4 years ago

Summary: Firefox does not play FLAC in Matroska → Add FLAC to webm

Phabricator Automation

Updated

•

4 years ago

Attachment #9119300 - Attachment is obsolete: true

Phabricator Automation

Updated

•

4 years ago

Attachment #9119301 - Attachment is obsolete: true