Last Comment Bug 476752 - support the speex voice codec in <audio> and <video> elements
: support the speex voice codec in <audio> and <video> elements
Status: VERIFIED WONTFIX
:
Product: Core
Classification: Components
Component: Audio/Video (show other bugs)
: Trunk
: All All
: -- enhancement with 10 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-02-03 17:22 PST by Ralph Giles (:rillian) needinfo me
Modified: 2015-02-23 04:28 PST (History)
25 users (show)
roc: wanted1.9.1-
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Add speex support (972.70 KB, patch)
2009-02-04 16:51 PST, cajbir (:cajbir)
no flags Details | Diff | Splinter Review

Description Ralph Giles (:rillian) needinfo me 2009-02-03 17:22:11 PST
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.04 (hardy) Firefox/3.0.5
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel MacOS X 10.4; en-US; rc:1.9.2a1pre) Gecko/20090120 Minefield/3.2a1pre

The Firefox 3.1 nightlies have support for Ogg media files under the html5 <audio> and <video> elements. Native browser support for multimedia is very exciting, and default inclusion of royalty-free formats is a great improvement in the usefulness of open media on the web.

Currently, the Ogg container is supported with the Vorbis audio codec and the Theora video codec. I request that support for the Speex voice codec be used as well. This codec is also royalty free, and similarly widely deployed to Vorbis. Speex has seen the most uptake in conferencing applications, where its bandwidth and latency advantages are most important. The latency difference isn't significant in the case of http streaming inside the browser, but the bandwidth is. Interviews, presentations, lectures, and other primarily talk-based segments are a significant part of web video, especially for users on low-bandwidth links.

The libfishsound library currently used to provide ogg support in mozilla already supports the libspeex reference implementation of this codec. So this request requires only a change to the build and review burden, not any new code.

Reproducible: Always

Steps to Reproduce:
1.Visit http://people.xiph.org/~giles/2009/speex.html
2.Click the play button

Actual Results:  
No audio plays.

Expected Results:  
Audio should play.
Comment 1 cajbir (:cajbir) 2009-02-03 17:24:42 PST
I'd also like to see Speex included. In the very early days I did include it, but removed it to reduce the footprint and need to analyse the code. 

Also note that Adobe Flash 10 includes Speex support.
Comment 2 Ralph Giles (:rillian) needinfo me 2009-02-03 17:34:52 PST
Just a follow up about the bandwidth advantages. The talk linked above is a 3.6 MB file. The corresponding vorbis version is 19 MB. A factor of 5 difference in bitrate.
Comment 3 cajbir (:cajbir) 2009-02-04 16:51:56 PST
Created attachment 360621 [details] [diff] [review]
Add speex support

Ralph Giles sent me a patch that did all the hard work to add libspeex to the build system. I did a couple of tweaks to the Ogg decoder and video element and this patch has speex working.

Stll to be done is to split the patch up into the 'libspeex' portion and a separate patch for the Mozilla changes for review.
Comment 4 Robert O'Callahan (:roc) (email my personal email if necessary) 2009-02-08 18:52:14 PST
What kind of fuzz testing has speex had?
Comment 5 Robert O'Callahan (:roc) (email my personal email if necessary) 2009-02-08 19:58:14 PST
We will also need internal legal analysis.
Comment 6 Ralph Giles (:rillian) needinfo me 2009-02-08 21:38:58 PST
Yay more legal analysis! :)

Note the patch contains some extra source files which aren't needed just for the decoder library. The makefile knows what's relevant.
Comment 7 Jean-Marc Valin (:jmspeex) 2009-02-08 21:57:02 PST
Speex was tested with Coverity, which revealed nothing to be concerned about (nothing security-related). A few bugs in the tools (not in libspeex), a few minor issues when functions are used incorrectly, and one "real" bug in the coded. That only "real" bug only affected the fixed-point version of the encoder and would only cause a quality degradation (not security-related). The only security issue with Speex I can remember was a case where apps assumed that the header parsing function did some validation when it did not. No bit-stream has ever been found to cause any security issue -- mostly because almost all bit sequences are actually *valid* Speex packets.
Comment 8 Gregory Maxwell 2009-02-08 22:28:21 PST
Around 2008-02-09 I ran on the order of several (tens of?) million fuzz testing cycles on Speex. This resulted in the discovery of a bug in speexdec (fixed in Xiph SVN >=r14512) but not the speex library itself.  Random fuzz testing gets reasonably deep penetration for Speex (moreso than for Vorbis, and FLAC), so I believe that it has had reasonable fuzz coverage.  

libspeex has also been through Coverity as recently as 2008-08-14 (r15180) with zero outstanding 'defects'.
Comment 9 Robert O'Callahan (:roc) (email my personal email if necessary) 2009-02-08 22:57:15 PST
That sounds good.
Comment 10 Robert O'Callahan (:roc) (email my personal email if necessary) 2009-02-10 10:38:20 PST
I'm going to mark this not wanted for 1.9.1 just because time is very short and legal alone is likely to make this impossible for 1.9.1.

Post-1.9.1, would it make more sense to focus on Celt instead of Speex?
Comment 11 Ralph Giles (:rillian) needinfo me 2009-02-10 11:19:20 PST
Thanks for the priority update.

I don't think Celt makes sense over Speex. Speex is very mature and still competitive for voice. We're very excited about Celt, but (a) it's still under development and not yet suitable for deployment outside custom applications, and (b) the goal is to be competitive with vorbis in quality/bitrate for general purpose audio but at much lower latency. While it's getting comparable at higher bitrates, we're a long way from (b) at very low bitrates. But more importantly, latency just isn't that important for http streaming on the html media element model.

If mozilla ever implements two-way (conferencing) media support, then it will be useful. This is a logical step in the browser-as-os plan, and a parity with flash checkbox, but I think there are lot of open issues from an api design and security perspective, and it makes more sense to develop conferencing support as an extension.

Other free codecs of interest are FLAC, which is widely used for lossless audio, and Dirac, which will (eventually) offer better quality/bitrate than theora, especially for HD content. I would suggest these as the next most useful additions.
Comment 12 Jean-Marc Valin (:jmspeex) 2009-02-10 11:21:42 PST
"Post-1.9.1, would it make more sense to focus on Celt instead of Speex?"

No. (not counting the fact that the CELT bit-stream isn't frozen yet) CELT is designed for very low delay (<10 ms) transmission, which (AFAIK) Mozilla doesn't care about because we're talking about pre-recorded files. So when you don't actually care about delay, then Vorbis will almost always give you better quality than CELT at the same bit-rate. The reason for including Speex is that for voice, it can give you better quality than Vorbis at low bit-rate. For example, you can have intelligible speech quality at 4 kbit/s (VBR) and below. You can find some samples at http://www.speex.org/samples/ to get an idea. I see Speex being useful mostly when there's either no video, or when the video is encoded at low bit-rate.
Comment 13 Robert O'Callahan (:roc) (email my personal email if necessary) 2009-02-10 11:39:57 PST
OK. Transmitting video and audio from the browser is something we're interested in, and I don't think the API and security issues are hard, but we can ship CELT and Speex if one doesn't dominate the other.
Comment 14 d 2010-05-05 10:27:08 PDT
I can see that not much has happened here since early 2009. Is this something we still want to ship? It's a good time to get it on-board the 1.9.3 release now.
Comment 15 cajbir (:cajbir) 2010-05-05 17:28:46 PDT
It has unfortunately bitrot substantially. We no longer use liboggplay, libfishsound or liboggz which this patch utilizes.
Comment 16 [Baboo] 2010-06-14 18:17:11 PDT
So, no Speex for 1.9.3?
Comment 17 cajbir (:cajbir) 2010-06-14 18:48:21 PDT
No one is actively working to get the patch updated to latest trunk afaik. I've removed myself from assignee so someone can take over if they want.
Comment 18 cajbir (:cajbir) 2011-02-28 20:11:39 PST
With the arrival of WebM there are no plans to support additional codecs within the Ogg container. Thanks for the efforts that you've put into the original patch.
Comment 19 [Baboo] 2011-02-28 23:55:43 PST
The WONTFIX is referring to Speex being implemented for the Ogg container format, not support for the codecs itself?
Comment 20 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-03-01 01:09:53 PST
The former, but given the existence of Opus the use-cases for Speex are getting very narrow, so I'm dubious about supporting it in any container.
Comment 21 John Villar 2011-06-29 09:42:26 PDT
Why won't support the Speex codec Robert? Opus is still emerging as a standard but has to consolidate itself yet. I know Speex could help a lot to lower bandwidth requirements for translations and games if we get support at least on firefox and chrome.
Comment 22 [Baboo] 2011-06-29 10:25:02 PDT
Concerning Opus, I wonder if the fact that in the meantime Skype has been bought by Microsoft will have an effect on its progress.
Comment 23 Timothy B. Terriberry (:derf) 2011-06-29 10:48:08 PDT
(In reply to comment #22)
> Concerning Opus, I wonder if the fact that in the meantime Skype has been
> bought by Microsoft will have an effect on its progress.

We've been working closely with Skype, even after the MS acquisition, and I have seen no slow-down or change in their commitment to get Opus done.
Comment 24 John Villar 2011-06-29 14:58:15 PDT
Still.... why not support an open standard in the audio tag? it seems to me that there's something against using speex as a possible format for the Audio tag, but why?
Comment 25 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-06-29 19:29:43 PDT
Because Opus will make it obsolete. There is almost no chance that'd you'd be able to use Speex in a real cross-browser application before Opus is ready for use.
Comment 26 John Villar 2011-06-30 05:39:42 PDT
(In reply to comment #25)
> Because Opus will make it obsolete. There is almost no chance that'd you'd
> be able to use Speex in a real cross-browser application before Opus is
> ready for use.
Still, Opus looks like it will take some time to mature (tests, stream changes, etc), instead Speex is a codec that has been around for quite some time already and has settled, and it is really good for what it is intended for (audio speech streams).

It wouldn't hurt if the audio tag had a way to support extended audio formats without recompiling the whole browser (like plugins).

And if Opus will make Speex obsolete, why you have MP3 support yet? isn't OGG/Vorbis way better and less encumbered?
Comment 27 cajbir (:cajbir) 2011-06-30 06:44:03 PDT
(In reply to comment #26)
> And if Opus will make Speex obsolete, why you have MP3 support yet? isn't
> OGG/Vorbis way better and less encumbered?

We don't support MP3 and we do support Vorbis in the audio element.
Comment 28 John Villar 2011-06-30 07:00:02 PDT
(In reply to comment #27)
> We don't support MP3 and we do support Vorbis in the audio element.

Sorry, you're right, that's chrome's case :-/

Ok, then a question: If there's no plans to support Speex codec (or any other voice specific codec for the meantime), can we game developers expect Opus to be implemented in the near future?

Remember games have two important use cases for voice-only audio:
* Non player character's dialogs
* Player to Player communication through speech

And this is one of the few features that's blocking full fledged games from becoming a reality for the web (the other ones being Mouse Lock API which is a proposal Vincent Scheib from google is working on and the Full screen API, which is also being actively worked).
Comment 29 [Baboo] 2011-06-30 11:38:07 PDT
(In reply to comment #25)
> There is almost no chance that'd you'd
> be able to use Speex in a real cross-browser application before Opus is
> ready for use.

Did any of the other browser vendors made (official?) indications towards supporting Opus?
Comment 30 Alex Vincent [:WeirdAl] 2011-07-25 23:55:48 PDT
I can't find a bug filed for Opus... and I don't know enough about it to file a good one.
Comment 31 John Drinkwater (:beta) 2011-07-26 07:47:27 PDT
(In reply to comment #28)
> (In reply to comment #27)
> > We don't support MP3 and we do support Vorbis in the audio element.
> 
> Sorry, you're right, that's chrome's case :-/
> 
> Ok, then a question: If there's no plans to support Speex codec (or any
> other voice specific codec for the meantime), can we game developers expect
> Opus to be implemented in the near future?

Opus has only just closed last call[1] for draft 7[2] at the IETF-codec working group, it is unlikely that it would be integrated in the  near future.

[1] http://www.ietf.org/mail-archive/web/codec/current/msg02584.html
[2] http://www.ietf.org/id/draft-ietf-codec-opus-07.txt
Comment 32 John Drinkwater (:beta) 2011-07-26 07:54:32 PDT
Made Bug 674225 to move Opus comments to there, apologies for bugspam.

Note You need to log in before you can comment on or make changes to this bug.