Closed Bug 749522 (webrtc-tests) Opened 10 years ago Closed 5 years ago

[meta] Tracking bug for WebRTC Automation

Categories

(Core :: WebRTC, defect, P2)

defect

Tracking

()

RESOLVED FIXED
mozilla35

People

(Reporter: jesup, Assigned: drno)

References

(Depends on 2 open bugs, )

Details

(Keywords: meta, Whiteboard: [WebRTC], [blocking-webrtc-])

We need tests for WebRTC.  Note that not every test discussed here will be in the first release.

We have imported test code already in the tree for sipcc (media/webrtc/signaling), and for webrtc (media/webrtc/trunk). Any of this existing test code we can enable is very useful (note that some might take a while to run, so we might limit it a bit or run certain tests only on demand or occasionally).  Also, gtest has been imported and can be used to enable existing tests that use it, or to write new tests where appropriate.

In addition, we're going to want to add full-up tests of the stack as it runs in the browser.  Much of that testing would be by manipulating getUserMedia() & MediaStreams and PeerConnection.  We'll want to leverage the ability to substitute fake device inputs and outputs, or to encode a MediaStream fed from a file.  I'm a little leery of using an encoded video file via a <video> element, in case the output stream from captureStreamUntilEnded isn't 100% predictable; I think it would be better to inject YUV and raw audio directly into source MediaStreams.

Some aspects of PeerConnections will be hard to test in our normal framework, unless we build some type of standalone WebRTC client we can talk to, or if it's possible to run two browsers (and it may be; that would be preferable).

For signaling, we'll want to test the generated SDP for a bunch of cases matches what we expect (a string compare would do, but would require too much updating to match idiosyncrasies of the generation - probably we should use regexps and partial parsing to check things like correct number of streams, which ones have port 0, etc.

We'll also need end-to-end DataChannel tests via PeerConnection.  This should cover basic connection and data transfer, reliable and unreliable channels, creation 'glare' (simultaneous creation), forcing an increase in the number of active streams, re-use of streams, behavior of a not-open-yet or closed data channel, etc).  Note that testing unreliable channels' unreliability will be a pain unless we go through a purposely lossy software proxy/router.

We should test putting streams on hold, resuming, adding, removing, multiple video and audio sources, etc.

And as usual, as we find and fix bugs we'll want to add tests for them where possible.

Additional suggestions and pointers welcome.
(In reply to Randell Jesup [:jesup] from comment #0)
> I'm a little leery of using an encoded video
> file via a <video> element, in case the output stream from
> captureStreamUntilEnded isn't 100% predictable;

What sort of unpredictability are you worried about?
(In reply to Randell Jesup [:jesup] from comment #0)
> We have imported test code already in the tree for sipcc
> (media/webrtc/signaling), and for webrtc (media/webrtc/trunk). Any of this
> existing test code we can enable is very useful (note that some might take a
> while to run, so we might limit it a bit or run certain tests only on demand
> or occasionally).  Also, gtest has been imported and can be used to enable

The other problem is that a number of them rely on large, raw YUV files, which I don't believe we've imported, so that will limit some of the tests we can run.
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #1)
> (In reply to Randell Jesup [:jesup] from comment #0)
> > I'm a little leery of using an encoded video
> > file via a <video> element, in case the output stream from
> > captureStreamUntilEnded isn't 100% predictable;
> 
> What sort of unpredictability are you worried about?

If a <video> element can skip or repeat frames based on timing or modify the bits in any way, the resultant stream might not be bit-repeatable (or might miss covering some input edge cases - invalid YUV, maximal saturation, evil patterns, etc.

Mostly this is just me being paranoid about lossy/adaptive encode/decode paths; I've seen enough oranges on tbpl that spring from timing issues that I'm wary...  If we can guarantee bit-exact decodings (including across decoder updates - again, likely so), then we're probably ok.

(In reply to Timothy B. Terriberry (:derf) from comment #2)
> The other problem is that a number of them rely on large, raw YUV files,
> which I don't believe we've imported, so that will limit some of the tests
> we can run.

I purposely kept them out; they could be added back in if we want. hg doesn't like 40MB files, though.  Or maybe there's a way to store them outside of the normal build tree.
Depends on: 749577
Depends on: dc-tests
QA Contact: jsmith
Keywords: meta
Whiteboard: [WebRTC], [blocking-webrtc+]
The automation development team will chime in here in the next days so that we can start to build up a suite of smoketests for the initial landing of the WebRTC feature. Right now I'm collecting all the necessary information to get started. Therefore I have created a project page which I will put into the URL field of this bug. It will be filled with details today and tomorrow.
Depends on: 781534
Per today's automation meeting, I'm spliting off the tracking of gUM tracking bug separate from the full webrtc stack.
No longer depends on: 781534
Depends on: pc-tests
Per triage, not blocking on the meta bug. Blocking on the individual test implementation bugs.
Whiteboard: [WebRTC], [blocking-webrtc+] → [WebRTC], [blocking-webrtc-]
Depends on: 781534
Summary: WebRTC tests needed → [meta] Create automated tests to cover smoke tests for features in the WebRTC stack
Clearing in-testsuite flag - let's track using this flag on individual bugs, not meta bugs.
Flags: in-testsuite?
Summary: [meta] Create automated tests to cover smoke tests for features in the WebRTC stack → [meta] Tracking bug for WebRTC Automation
Depends on: 822109
Alias: webrtc-tests
Depends on: 780805
Depends on: 810350
Depends on: 815002
Depends on: 862883
No longer depends on: 810350
No longer depends on: 780805
Depends on: 867778
Depends on: 868035
Depends on: 908473
Blocks: 970426
QA Contact: jsmith → drno
No longer blocks: 970426
needs to break down for what we want to do in Fx33 vrs Fx34
Priority: -- → P2
Target Milestone: --- → mozilla33
Nils -- Does this bug need more bug breakdown or are we good?
Assignee: nobody → drno
Flags: needinfo?(drno)
From my perspective this is a meta bug tracking all the WebRTC test we ever want to implement. As we won't get close to get this done in one of the upcoming releases I would suggest to not link this bug as a dependency for delivery.
A better option is probably to create a test meta bug per release to track what is still missing per release.
Flags: needinfo?(drno)
Target Milestone: mozilla33 → mozilla35
Resolved - replaced with new meta: See Bug 1343560
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.