Closed Bug 476752 Opened 16 years ago Closed 14 years ago

support the speex voice codec in <audio> and <video> elements

Categories

(Core :: Audio/Video, enhancement)

Product:
Core
Component:
Audio/Video
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED WONTFIX

People

(Reporter: rillian, Unassigned)

Details

Attachments

(1 file)

Add speex support
972.70 KB, patch
Details | Diff | Splinter Review
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.04 (hardy) Firefox/3.0.5 Build Identifier: Mozilla/5.0 (Macintosh; U; Intel MacOS X 10.4; en-US; rc:1.9.2a1pre) Gecko/20090120 Minefield/3.2a1pre The Firefox 3.1 nightlies have support for Ogg media files under the html5 <audio> and <video> elements. Native browser support for multimedia is very exciting, and default inclusion of royalty-free formats is a great improvement in the usefulness of open media on the web. Currently, the Ogg container is supported with the Vorbis audio codec and the Theora video codec. I request that support for the Speex voice codec be used as well. This codec is also royalty free, and similarly widely deployed to Vorbis. Speex has seen the most uptake in conferencing applications, where its bandwidth and latency advantages are most important. The latency difference isn't significant in the case of http streaming inside the browser, but the bandwidth is. Interviews, presentations, lectures, and other primarily talk-based segments are a significant part of web video, especially for users on low-bandwidth links. The libfishsound library currently used to provide ogg support in mozilla already supports the libspeex reference implementation of this codec. So this request requires only a change to the build and review burden, not any new code. Reproducible: Always Steps to Reproduce: 1.Visit http://people.xiph.org/~giles/2009/speex.html 2.Click the play button Actual Results: No audio plays. Expected Results: Audio should play.
I'd also like to see Speex included. In the very early days I did include it, but removed it to reduce the footprint and need to analyse the code. Also note that Adobe Flash 10 includes Speex support.
Just a follow up about the bandwidth advantages. The talk linked above is a 3.6 MB file. The corresponding vorbis version is 19 MB. A factor of 5 difference in bitrate.
OS: Linux → All
Hardware: x86 → All
Version: unspecified → Trunk
Attached patch Add speex support — — Splinter Review
Ralph Giles sent me a patch that did all the hard work to add libspeex to the build system. I did a couple of tweaks to the Ogg decoder and video element and this patch has speex working. Stll to be done is to split the patch up into the 'libspeex' portion and a separate patch for the Mozilla changes for review.
Assignee: nobody → chris.double
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: wanted1.9.1?
What kind of fuzz testing has speex had?
We will also need internal legal analysis.
Yay more legal analysis! :) Note the patch contains some extra source files which aren't needed just for the decoder library. The makefile knows what's relevant.
Speex was tested with Coverity, which revealed nothing to be concerned about (nothing security-related). A few bugs in the tools (not in libspeex), a few minor issues when functions are used incorrectly, and one "real" bug in the coded. That only "real" bug only affected the fixed-point version of the encoder and would only cause a quality degradation (not security-related). The only security issue with Speex I can remember was a case where apps assumed that the header parsing function did some validation when it did not. No bit-stream has ever been found to cause any security issue -- mostly because almost all bit sequences are actually *valid* Speex packets.
Around 2008-02-09 I ran on the order of several (tens of?) million fuzz testing cycles on Speex. This resulted in the discovery of a bug in speexdec (fixed in Xiph SVN >=r14512) but not the speex library itself. Random fuzz testing gets reasonably deep penetration for Speex (moreso than for Vorbis, and FLAC), so I believe that it has had reasonable fuzz coverage. libspeex has also been through Coverity as recently as 2008-08-14 (r15180) with zero outstanding 'defects'.
I'm going to mark this not wanted for 1.9.1 just because time is very short and legal alone is likely to make this impossible for 1.9.1. Post-1.9.1, would it make more sense to focus on Celt instead of Speex?
Flags: wanted1.9.1?
Thanks for the priority update. I don't think Celt makes sense over Speex. Speex is very mature and still competitive for voice. We're very excited about Celt, but (a) it's still under development and not yet suitable for deployment outside custom applications, and (b) the goal is to be competitive with vorbis in quality/bitrate for general purpose audio but at much lower latency. While it's getting comparable at higher bitrates, we're a long way from (b) at very low bitrates. But more importantly, latency just isn't that important for http streaming on the html media element model. If mozilla ever implements two-way (conferencing) media support, then it will be useful. This is a logical step in the browser-as-os plan, and a parity with flash checkbox, but I think there are lot of open issues from an api design and security perspective, and it makes more sense to develop conferencing support as an extension. Other free codecs of interest are FLAC, which is widely used for lossless audio, and Dirac, which will (eventually) offer better quality/bitrate than theora, especially for HD content. I would suggest these as the next most useful additions.
"Post-1.9.1, would it make more sense to focus on Celt instead of Speex?" No. (not counting the fact that the CELT bit-stream isn't frozen yet) CELT is designed for very low delay (<10 ms) transmission, which (AFAIK) Mozilla doesn't care about because we're talking about pre-recorded files. So when you don't actually care about delay, then Vorbis will almost always give you better quality than CELT at the same bit-rate. The reason for including Speex is that for voice, it can give you better quality than Vorbis at low bit-rate. For example, you can have intelligible speech quality at 4 kbit/s (VBR) and below. You can find some samples at http://www.speex.org/samples/ to get an idea. I see Speex being useful mostly when there's either no video, or when the video is encoded at low bit-rate.
OK. Transmitting video and audio from the browser is something we're interested in, and I don't think the API and security issues are hard, but we can ship CELT and Speex if one doesn't dominate the other.
I can see that not much has happened here since early 2009. Is this something we still want to ship? It's a good time to get it on-board the 1.9.3 release now.
It has unfortunately bitrot substantially. We no longer use liboggplay, libfishsound or liboggz which this patch utilizes.
So, no Speex for 1.9.3?
No one is actively working to get the patch updated to latest trunk afaik. I've removed myself from assignee so someone can take over if they want.
Assignee: chris.double → nobody
With the arrival of WebM there are no plans to support additional codecs within the Ogg container. Thanks for the efforts that you've put into the original patch.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → WONTFIX
The WONTFIX is referring to Speex being implemented for the Ogg container format, not support for the codecs itself?
The former, but given the existence of Opus the use-cases for Speex are getting very narrow, so I'm dubious about supporting it in any container.
Status: RESOLVED → VERIFIED
Why won't support the Speex codec Robert? Opus is still emerging as a standard but has to consolidate itself yet. I know Speex could help a lot to lower bandwidth requirements for translations and games if we get support at least on firefox and chrome.
Concerning Opus, I wonder if the fact that in the meantime Skype has been bought by Microsoft will have an effect on its progress.
(In reply to comment #22) > Concerning Opus, I wonder if the fact that in the meantime Skype has been > bought by Microsoft will have an effect on its progress. We've been working closely with Skype, even after the MS acquisition, and I have seen no slow-down or change in their commitment to get Opus done.
Still.... why not support an open standard in the audio tag? it seems to me that there's something against using speex as a possible format for the Audio tag, but why?
Because Opus will make it obsolete. There is almost no chance that'd you'd be able to use Speex in a real cross-browser application before Opus is ready for use.
(In reply to comment #25) > Because Opus will make it obsolete. There is almost no chance that'd you'd > be able to use Speex in a real cross-browser application before Opus is > ready for use. Still, Opus looks like it will take some time to mature (tests, stream changes, etc), instead Speex is a codec that has been around for quite some time already and has settled, and it is really good for what it is intended for (audio speech streams). It wouldn't hurt if the audio tag had a way to support extended audio formats without recompiling the whole browser (like plugins). And if Opus will make Speex obsolete, why you have MP3 support yet? isn't OGG/Vorbis way better and less encumbered?
(In reply to comment #26) > And if Opus will make Speex obsolete, why you have MP3 support yet? isn't > OGG/Vorbis way better and less encumbered? We don't support MP3 and we do support Vorbis in the audio element.
(In reply to comment #27) > We don't support MP3 and we do support Vorbis in the audio element. Sorry, you're right, that's chrome's case :-/ Ok, then a question: If there's no plans to support Speex codec (or any other voice specific codec for the meantime), can we game developers expect Opus to be implemented in the near future? Remember games have two important use cases for voice-only audio: * Non player character's dialogs * Player to Player communication through speech And this is one of the few features that's blocking full fledged games from becoming a reality for the web (the other ones being Mouse Lock API which is a proposal Vincent Scheib from google is working on and the Full screen API, which is also being actively worked).
(In reply to comment #25) > There is almost no chance that'd you'd > be able to use Speex in a real cross-browser application before Opus is > ready for use. Did any of the other browser vendors made (official?) indications towards supporting Opus?
I can't find a bug filed for Opus... and I don't know enough about it to file a good one.
(In reply to comment #28) > (In reply to comment #27) > > We don't support MP3 and we do support Vorbis in the audio element. > > Sorry, you're right, that's chrome's case :-/ > > Ok, then a question: If there's no plans to support Speex codec (or any > other voice specific codec for the meantime), can we game developers expect > Opus to be implemented in the near future? Opus has only just closed last call[1] for draft 7[2] at the IETF-codec working group, it is unlikely that it would be integrated in the near future. [1] http://www.ietf.org/mail-archive/web/codec/current/msg02584.html [2] http://www.ietf.org/id/draft-ietf-codec-opus-07.txt
Made Bug 674225 to move Opus comments to there, apologies for bugspam.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: