749522 - (webrtc-tests) [meta] Tracking bug for WebRTC Automation

Reporter

Description

•

12 years ago

We need tests for WebRTC. Note that not every test discussed here will be in the first release.

We have imported test code already in the tree for sipcc (media/webrtc/signaling), and for webrtc (media/webrtc/trunk). Any of this existing test code we can enable is very useful (note that some might take a while to run, so we might limit it a bit or run certain tests only on demand or occasionally). Also, gtest has been imported and can be used to enable existing tests that use it, or to write new tests where appropriate.

In addition, we're going to want to add full-up tests of the stack as it runs in the browser. Much of that testing would be by manipulating getUserMedia() & MediaStreams and PeerConnection. We'll want to leverage the ability to substitute fake device inputs and outputs, or to encode a MediaStream fed from a file. I'm a little leery of using an encoded video file via a <video> element, in case the output stream from captureStreamUntilEnded isn't 100% predictable; I think it would be better to inject YUV and raw audio directly into source MediaStreams.

Some aspects of PeerConnections will be hard to test in our normal framework, unless we build some type of standalone WebRTC client we can talk to, or if it's possible to run two browsers (and it may be; that would be preferable).

For signaling, we'll want to test the generated SDP for a bunch of cases matches what we expect (a string compare would do, but would require too much updating to match idiosyncrasies of the generation - probably we should use regexps and partial parsing to check things like correct number of streams, which ones have port 0, etc.

We'll also need end-to-end DataChannel tests via PeerConnection. This should cover basic connection and data transfer, reliable and unreliable channels, creation 'glare' (simultaneous creation), forcing an increase in the number of active streams, re-use of streams, behavior of a not-open-yet or closed data channel, etc). Note that testing unreliable channels' unreliability will be a pain unless we go through a purposely lossy software proxy/router.

We should test putting streams on hold, resuming, adding, removing, multiple video and audio sources, etc.

And as usual, as we find and fix bugs we'll want to add tests for them where possible.

Additional suggestions and pointers welcome.

Robert O'Callahan (:roc) (email my personal email if necessary)

Comment 1

•

12 years ago

(In reply to Randell Jesup [:jesup] from comment #0)
> I'm a little leery of using an encoded video
> file via a <video> element, in case the output stream from
> captureStreamUntilEnded isn't 100% predictable;

What sort of unpredictability are you worried about?

Timothy B. Terriberry (:derf)

Comment 2

•

12 years ago

(In reply to Randell Jesup [:jesup] from comment #0)
> We have imported test code already in the tree for sipcc
> (media/webrtc/signaling), and for webrtc (media/webrtc/trunk). Any of this
> existing test code we can enable is very useful (note that some might take a
> while to run, so we might limit it a bit or run certain tests only on demand
> or occasionally).  Also, gtest has been imported and can be used to enable

The other problem is that a number of them rely on large, raw YUV files, which I don't believe we've imported, so that will limit some of the tests we can run.

Randell Jesup [:jesup] (needinfo me)

Reporter

Comment 3

•

12 years ago

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #1)
> (In reply to Randell Jesup [:jesup] from comment #0)
> > I'm a little leery of using an encoded video
> > file via a <video> element, in case the output stream from
> > captureStreamUntilEnded isn't 100% predictable;
> 
> What sort of unpredictability are you worried about?

If a <video> element can skip or repeat frames based on timing or modify the bits in any way, the resultant stream might not be bit-repeatable (or might miss covering some input edge cases - invalid YUV, maximal saturation, evil patterns, etc.

Mostly this is just me being paranoid about lossy/adaptive encode/decode paths; I've seen enough oranges on tbpl that spring from timing issues that I'm wary...  If we can guarantee bit-exact decodings (including across decoder updates - again, likely so), then we're probably ok.

(In reply to Timothy B. Terriberry (:derf) from comment #2)
> The other problem is that a number of them rely on large, raw YUV files,
> which I don't believe we've imported, so that will limit some of the tests
> we can run.

I purposely kept them out; they could be added back in if we want. hg doesn't like 40MB files, though.  Or maybe there's a way to store them outside of the normal build tree.

Randell Jesup [:jesup] (needinfo me)

Reporter