720139 - The Search for A Low Latency Encoder

Status: RESOLVED INCOMPLETE

(Air Mozilla :: Streaming, defect)

Description

[Richard A Milewski[:richard]]

A high-quality, low-latency encoder would make it possible to eliminate many of the expensive, closed proprietary systems in MozSpaces and video conference rooms.
It would enable browser-based control of 10Fwd and similar MozSpaces instead of relying on complex and expensive matrix switches which have proprietary interfaces.

Comment 1

[Richard A Milewski[:richard]]

Extron offers a solution in the AV-Matrix 225 series of hardware codecs.  

http://

Although these offer 35ms latency, and superb image quality, there are problems that probably eliminate them from consideration:

1 - Prohibitively High Cost ($8K - $13K per codec)   

2 - Proprietary video format (Pure3) which would work for internal switching but is not suitable for final streaming to Mozillians and the public.  (Is it even important to have the same video format for both uses?)

Comment 2

[Richard A Milewski[:richard]]

Link to Extron VN-Matrix 225 data sheets: 

http://www.extron.com/product/product.aspx?id=vnmatrix225&search=AV-Matrix%20225

Comment 3

[Richard A Milewski[:richard]]

http://x264dev.multimedia.cx/archives/249

It appears from the blog post linked above that X.264 can be configured for low latency (Possibly lower than the Extron Pure3 codecs).  Testing needed.

Can similar things be done in VP8/WebM?

Comment 4

[Ralph Giles (:rillian)]

The libvpx webm encoder has a real-time mode and adjustable latency, but currently requires quite hefty hardware to run at high quality. This may be more feasible if our xvp8 encoder project bears fruit later in 2012. (https://github.com/DarkShikari/xvp8)

If we want a solution just for internal switching, the BBC codec from diracvideo.org would be a better choice; the 'Dirac Pro' subset (also known at VC-2) is designed for low latency distribution in production work. This is also an open source/royalty free format aligned with our values.

Comment 5

[David Richards]

Krad Radio supports dirac, and will support xvp8 as soon as I get word it can produce a valid stream. When I say supports, it can encode and decode and play/write/read dirac from an MKV stream, its not done, but this is something I am very interested in doing.

Comment 6

[David Schleef]

This is due to buffering in Firefox.  Here's where latency comes from, rough numbers:

  70 ms in camera
  30 to 150 ms in the encoder
  30-1000 ms (configurable) for constant bit rate buffering
  3000 ms buffering in Firefox playback

This is all configured for clean playback in worst-case situations, which is appropriate for some use cases, but probably not this one.  For live video, Flash is configured for average case, which means that it will occasionally have video dropouts for few frames, and sometimes audio skipping.  It also means that it achieves much better average-case latency.  It does this by:

  70 ms in the camera
  30-70 ms in the encoder
  0 constant bit rate buffering (it assumes you have infinite bandwidth)
  <30 ms in playback

As I said, the tradeoff is skipping in playback, giving the "webcam teleconferencing" effect.

We can easily do 200 ms end-to-end (worst-case) latency with an RTSP client, or with client configured for unbuffered HTTP.  This is with stock libvpx or libtheora (btw, libtheora can do this partly due to work done by rillian).  Trying to get too much under 150 ms requires switching to much more expensive line-based cameras, line-based capture, and encoding techniques.

Schroedinger (the Dirac implementation) has the ability to due really tight control of CBR buffering and real-time encoding down to about 30 ms, but this is technically not necessary when doing average-case 200-ms-latency streaming.

Comment 7

[Ralph Giles (:rillian)]

As far as making this practical, we could do "unbuffered" http streaming in Firefox with a new DOM api to control the media element's buffering. This isn't too big of a patch, but would be Firefox-specific. As such the platform team has no enthusiasm for it.

WebRTC is an open standards effort which will give us equivalent latency to an RTSP client in the browser. That's what the platform team would like to support for conferencing and latency-sensitive webcasts. It will be 2013 before a useful version of that is deployed in Firefox, Chrome, and possibly other browsers.

We can explore other options for conferencing and presentations in the meantime, I think WebRTC should be the ultimate resolution of the bug.

Comment 8

[Mardeg]

On the audio side, bug 743720 implemented the Speex resampler, should we also take the "acoustic echo cancellation with multiple microphones and multiple loudspeakers" from the libspeexdsp source at http://www.speex.org/downloads/ ?

Comment 9

[Ralph Giles (:rillian)]

(In reply to Mardeg from comment #8)
> should we also take the "acoustic echo cancellation with multiple microphones and
> multiple loudspeakers" from the libspeexdsp source at http://www.speex.org/downloads/ ?

Can you be more specific about how that would help?

The WebRTC code we're intending to land does include an echo canceller. Doing that with stereo (or surround) is the hard part, but that should probably a different bug; it's pretty far out on the roadmap.

Comment 10

[Richard A Milewski[:richard]]

Closing this.   We'll be doing the first WebRTC experiments in Air Mozilla this fall.  We plan to use it for self-serve New Hire recordings.

More uses will follow... notably experiments with showing the multi-view outputs from the ATEMs in remote locations.