1295352 - Update media tests to not leave active gUM streams behind.

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

The getUserMedia API returns a stream with one or two tracks in it, yet requires users to call .stop() on each of said tracks, as well as all clones of those tracks, before the hardware resources that were reserved are relinquished (to avoid them being held until GC).

This can get tricky, given that scripts may clone and move tracks between streams ad nauseam, and indeed a lot of our own tests lose track of their tracks.

We should fix those tests, because, while not calling .stop() is fine and should be tested like anything else, we ideally want our tests to be reproducible and explicit in what they target rather than test things by accident, so that when things fail or intermittents happen, they're easy to track down.

Also, concurrent gUM requests may produce different results in tests in response to constraints, compared to non-concurrent requests. Basically, we don't want test behavior to depend on or vary based on what other tests they follow, or fail intermittently because of GC (note that we still tear things down on navigation, so this problem is contained to multiple tests in the same file).

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1295352 - Check that there is no active gUM after each runTest test.

Review commit: https://reviewboard.mozilla.org/r/71714/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/71714/

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1295352 - Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards.

Review commit: https://reviewboard.mozilla.org/r/71716/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/71716/

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1295352 - Fix media mochitests to not end with active gUM.

Review commit: https://reviewboard.mozilla.org/r/71718/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/71718/

Comment 4

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8781292 [details]
Bug 1295352 - Check that there is no active gUM after each runTest test.

https://reviewboard.mozilla.org/r/71714/#review69360

What about peerConnection tests?

runTestWhenReady is in head.js and it seems to me like we could wrap the check into that.

::: dom/media/tests/mochitest/mediaStreamPlayback.js:262
(Diff revision 1)
> +  .then(([device]) => device &&
> +      is(device.label, "", "Test must leave no active gUM streams behind."));

Does this work because you with the pref just removed the real device, but if they're active they'll still show up?

Perhaps a comment to explain for the uninvited reader.

::: dom/media/tests/mochitest/test_getUserMedia_constraints.html:136
(Diff revision 1)
> -      .then(() => stream.getAudioTracks()[0].applyConstraints({ })))
> +      .then(() => stream.getAudioTracks()[0].applyConstraints({ }))
> +      .then(() => stream.getTracks().forEach(track => track.stop())))

Should this be in the last patch?

Comment 5

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8781293 [details]
Bug 1295352 - Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards.

https://reviewboard.mozilla.org/r/71716/#review69362

::: dom/media/tests/mochitest/mediaStreamPlayback.js:42
(Diff revision 1)
> +  playMedia : function(isResume) {
> +    return this.playMediaWithMediaStreamTracksStop(isResume);

Looks like an alias now. Should we rename playMediaWithMediaStreamTracksStop to playMedia?

::: dom/media/tests/mochitest/mediaStreamPlayback.js:163
(Diff revision 1)
>             "Current src should still be an empty string");
>        });
>    },
>  
>    /**
>     * Stops the media with the associated stream.

Update the comment.

Comment 6

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8781294 [details]
Bug 1295352 - Fix media mochitests to not end with active gUM.

https://reviewboard.mozilla.org/r/71718/#review69364

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Bumping priority since this may be leading to some of our intermittents and perf issues.  Likely doesn't matter as we have patches and r+'s

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Any idea why this patch causes the following intermittent on try?

> 19:00:12     INFO -  1937 INFO TEST-START | dom/media/tests/mochitest/test_getUserMedia_bug1223696.html
> 19:00:12     INFO -  ++DOMWINDOW == 21 (0x709fa400) [pid = 5581] [serial = 98] [outer = 0x9c06e000]
> 19:00:12     INFO -  TEST DEVICES: Using media devices:
> 19:00:12     INFO -  audio: Sine source at 440 Hz
> 19:00:12     INFO -  video: Dummy video device
> 19:00:13     INFO -  --DOMWINDOW == 20 (0x709e8000) [pid = 5581] [serial = 95] [outer = (nil)] [url = chrome://browser/content/webrtcIndicator.xul]
> 19:00:14     INFO -  --DOMWINDOW == 19 (0x70aeb800) [pid = 5581] [serial = 97] [outer = (nil)] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
> 19:00:14     INFO -  --DOMWINDOW == 18 (0x709c6400) [pid = 5581] [serial = 90] [outer = (nil)] [url = http://mochi.test:8888/tests/dom/media/tests/mochitest/test_getUserMedia_basicVideo_playAfterLoadedmetadata.html]
> 19:00:14     INFO -  --DOMWINDOW == 17 (0x709e8c00) [pid = 5581] [serial = 96] [outer = (nil)] [url = about:blank]
> 19:00:14     INFO -  [5581] WARNING: Cannot query channel count on a AudioSegment with no chunks.: '!mChunks.IsEmpty()', file /builds/slave/try-lx-d-000000000000000000000/build/src/dom/media/AudioSegment.h, line 369
> 19:00:14     INFO -  [5581] WARNING: No audio tracks, but full-duplex audio is enabled!!!!!: file /builds/slave/try-lx-d-000000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 505
> 19:00:14     INFO -  ++DOCSHELL 0x709d5400 == 9 [pid = 5581] [id = 27]
> 19:00:14     INFO -  ++DOMWINDOW == 18 (0x709d5c00) [pid = 5581] [serial = 99] [outer = (nil)]
> 19:00:14     INFO -  ++DOMWINDOW == 19 (0x709d9800) [pid = 5581] [serial = 100] [outer = 0x709d5c00]
> 19:00:16     INFO -  [5581] ###!!! ASSERTION: Track not found: 'Error', file /builds/slave/try-lx-d-000000000000000000000/build/src/dom/media/MediaStreamGraph.cpp, line 2857
> 19:01:29     INFO -  #01: mozilla::SourceMediaStream::GetEndOfAppendedData [dom/media/MediaStreamGraph.cpp:2858]
> 19:01:29     INFO -  #02: mozilla::MediaEngineRemoteVideoSource::NotifyPull [dom/media/webrtc/MediaEngineRemoteVideoSource.cpp:336]
> 19:01:29     INFO -  #03: mozilla::GetUserMediaCallbackMediaStreamListener::NotifyPull [dom/media/MediaManager.cpp:407]
> 19:01:29     INFO -  #04: mozilla::MediaStreamGraphImpl::ExtractPendingInput [xpcom/glue/Mutex.h:206]
> 19:01:29     INFO -  #05: mozilla::MediaStreamGraphImpl::UpdateGraph [dom/media/MediaStreamGraph.cpp:1246]
> 19:01:29     INFO -  #06: mozilla::MediaStreamGraphImpl::OneIteration [dom/media/MediaStreamGraph.cpp:1392]
> 19:01:29     INFO -  #07: mozilla::AudioCallbackDriver::DataCallback [dom/media/GraphDriver.cpp:924]
> 19:01:29     INFO -  #08: trigger_user_callback [media/libcubeb/src/cubeb_pulse.c:220]
> 19:01:29     INFO -  #09: stream_read_callback [media/libcubeb/src/cubeb_pulse.c:323]
> 19:01:29     INFO -  #10: libpulse.so.0 + 0xb627
> 19:01:29     INFO -  #11: ??? (???:???)
> 19:01:29     INFO -  --DOCSHELL 0x709d5400 == 8 [pid = 5581] [id = 27]
> 19:01:29     INFO -  MEMORY STAT | vsize 963MB | residentFast 207MB | heapAllocated 68MB
> 19:01:29     INFO -  1938 INFO TEST-OK | dom/media/tests/mochitest/test_getUserMedia_bug1223696.html | took 4978ms
> 19:01:29     INFO -  ++DOMWINDOW == 20 (0x709ed800) [pid = 5581] [serial = 101] [outer = 0x9c06e000]
> 19:01:29     INFO -  1939 INFO TEST-UNEXPECTED-ERROR | dom/media/tests/mochitest/test_getUserMedia_bug1223696.html | Assertion count 1 is greater than expected range 0-0 assertions.

Here are the changes to test_getUserMedia_bug1223696.html: https://reviewboard.mozilla.org/r/71718/diff/1#2

I'm stumped.

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hmm, linux only. We use loopback devices rather than fake ones there...

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8781292 [details]
Bug 1295352 - Check that there is no active gUM after each runTest test.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/71714/diff/1-2/

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8781293 [details]
Bug 1295352 - Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/71716/diff/1-2/

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8781294 [details]
Bug 1295352 - Fix media mochitests to not end with active gUM.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/71718/diff/1-2/

Comment 13

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

I saw it locally with real devices recently. Can try to repro and investigate, but will take a while before I get there.

Comment 14

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Though, I could imagine it being because stream.stop() ends the tracks at the SourceMediaStream before the MediaEngine has been notified.

Comment 15

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

We have ended the track so it's been removed at the source (causing this NS_ERROR if you go looking for it), but the stream is still going so we're still getting NotifyPull calls. mState is covered by mMonitor so we can check it in NotifyPull to avoid looking for the track, just like we do in MediaEngineDefault.

Comment 16

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1295352 - Check state in MediaEngines' NotifyPull().

GetEndOfAppendedData() returns null and calls a NS_ERROR() if the track we're
looking for doesn't exist - to indicate an error in the caller's code.

When we end a MediaEngine track we set the state to stopped, which we can use
to guard the calls to GetEndOfAppendedData() (and appending data in general).
The locking is already in place.

Review commit: https://reviewboard.mozilla.org/r/73540/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/73540/

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8783870 [details]
Bug 1295352 - Check state in MediaEngines' NotifyPull().

https://reviewboard.mozilla.org/r/73540/#review71420

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7e3e593e8141
Check state in MediaEngines' NotifyPull(). r=jesup

Comment 19

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry, I had to back this out for failing mda test test_getUserMedia_basicTabshare.html:

https://hg.mozilla.org/integration/autoland/rev/5bde9ecced2bce88700dec6f5a8beeeb98766774

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=2411861&repo=autoland

11:15:08     INFO -  1972 INFO TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_getUserMedia_basicTabshare.html | Error executing test: Error: verifyPlaying timed out timeout/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:463:63 ... Async*timeout@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:463:26 ... MediaStreamPlayback.prototype.verifyPlaying@http://mochi.test:8888/tests/dom/media/tests/mochitest/mediaStreamPlayback.js:126:12 ... Async*@http://mochi.test:8888/tests/dom/media/tests/mochitest/test_getUserMedia_basicTabshare.html:27:12 ... Async*runTestWhenReady@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:350:10 ... runTest/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/mediaStreamPlayback.js:258:15 ... Async*runTest@http://mochi.test:8888/tests/dom/media/tests/mochitest/mediaStreamPlayback.js:257:31 ... @http://mochi.test:8888/tests/dom/media/tests/mochitest/test_getUserMedia_basicTabshare.html:19:3 ...
11:15:08     INFO -      runTestWhenReady/<@dom/media/tests/mochitest/head.js:352:7
11:15:08     INFO -      Async*runTestWhenReady@dom/media/tests/mochitest/head.js:350:10
11:15:08     INFO -      Async*runTest@dom/media/tests/mochitest/mediaStreamPlayback.js:257:31
11:15:08     INFO -      @dom/media/tests/mochitest/test_getUserMedia_basicTabshare.html:19:3

Comment 20

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Oh my, tab sources don't use mState. Or end their tracks at all. I think it's a pretty simple fix.

Sorry for causing a backout aryx.

Comment 21

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1295352 - Use mState in MediaEngineTabVideoSource.

Review commit: https://reviewboard.mozilla.org/r/74136/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74136/

Comment 22

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Full try (incl bug 1294605): https://treeherder.mozilla.org/#/jobs?repo=try&revision=834a0de7ebc1

Comment 23

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8784811 [details]
Bug 1295352 - Use mState in MediaEngineTabVideoSource.

https://reviewboard.mozilla.org/r/74136/#review72004

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1a7948aa1a31
Check that there is no active gUM after each runTest test. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/93860eea4310
Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7c030a81408
Fix media mochitests to not end with active gUM. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/df194bbc13b6
Check state in MediaEngines' NotifyPull(). r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/65f1bf3e9dec
Use mState in MediaEngineTabVideoSource. r=jesup

Comment 25

[Iris Hsiao [:ihsiao]]

Sorry had to back out for Mochitest Media failures, e.g., https://treeherder.mozilla.org/logviewer.html#?job_id=34747845&repo=mozilla-inbound#L5288

Comment 26

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3f2b3cac63e4
Backed out changeset 65f1bf3e9dec for Mochitest Media failure
https://hg.mozilla.org/integration/mozilla-inbound/rev/f61ce6d91281
Backed out changeset df194bbc13b6 
https://hg.mozilla.org/integration/mozilla-inbound/rev/402acc60fa97
Backed out changeset a7c030a81408 
https://hg.mozilla.org/integration/mozilla-inbound/rev/9459c9c3c2c6
Backed out changeset 93860eea4310 
https://hg.mozilla.org/integration/mozilla-inbound/rev/e5a3913dec6f
Backed out changeset 1a7948aa1a31

Comment 27

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Sorry. Should have thought about bug 1259788 landing in between - it was not in the comment 22 try push.

Comment 28

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

About the error in comment 25:

test_getUserMedia_mediaElementCapture_audio.html never calls gUMTrack.stop(), leaving an unstopped audio track. [1]

The try error message:

  "Test must leave no active gUM streams behind. - got "Dummy video device (0x0000)", expected "" "

...is confusing, and doesn't mean a video track was leaked. Rather, the noGum test happens to use a trick to detect if devices returned from enumerateDevices include labels, using is(device.label, "", ...) on the first device listed, which is always a video device. It has no correlation with what media type was leaked, which there's no way to detect.

[1] https://dxr.mozilla.org/mozilla-central/rev/a551f534773cf2d6933f78ce7d82a7a33a99643e/dom/media/tests/mochitest/test_getUserMedia_mediaElementCapture_audio.html#96

Comment 29

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1295352 - Update mediaElementCapture tests to stop all tracks before finishing.

Review commit: https://reviewboard.mozilla.org/r/74982/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74982/

Comment 30

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8783870 [details]
Bug 1295352 - Check state in MediaEngines' NotifyPull().

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/73540/diff/1-2/

Comment 31

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8784811 [details]
Bug 1295352 - Use mState in MediaEngineTabVideoSource.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/74136/diff/1-2/

Comment 32

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a18a3b666cb

Comment 33

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8785971 [details]
Bug 1295352 - Update mediaElementCapture tests to stop all tracks before finishing.

https://reviewboard.mozilla.org/r/74982/#review72818

::: dom/media/tests/mochitest/test_getUserMedia_mediaElementCapture_video.html:114
(Diff revision 1)
>      gUMVideoElement.srcObject.addTrack(track);
>      gUMVideoElement.play();
>      return checkVideoPlaying(captureStreamElement);
>    })
> -  .then(() => ok(true, "Test passed."))
> +  .then(() => {
> +    gUMVideoElement.srcObject.getTracks().map(t => t.stop());

forEach?

Comment 34

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8785971 [details]
Bug 1295352 - Update mediaElementCapture tests to stop all tracks before finishing.

https://reviewboard.mozilla.org/r/74982/#review72820

Comment 35

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Attachment: Bug 1295352 - Release MediaEngineTabVideoSource main-thread-only members on main thread.

Review commit: https://reviewboard.mozilla.org/r/75308/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/75308/

Comment 36

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

Comment on attachment 8785971 [details]
Bug 1295352 - Update mediaElementCapture tests to stop all tracks before finishing.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/74982/diff/1-2/

Comment 37

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

New try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8816ecd1b73

Comment 38

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8786373 [details]
Bug 1295352 - Release MediaEngineTabVideoSource main-thread-only members on main thread.

https://reviewboard.mozilla.org/r/75308/#review73560

Take your pick how to handle it.  Add a comment that those are main-thread-only and thus the Dispatch (or rename to DestroyMainthreadRefs or some such)

::: dom/media/webrtc/MediaEngineTabVideoSource.cpp:207
(Diff revision 1)
>  
>  nsresult
>  MediaEngineTabVideoSource::Deallocate(AllocationHandle* aHandle)
>  {
>    MOZ_ASSERT(!aHandle);
> +  NS_DispatchToMainThread(do_AddRef(new DestroyRunnable(this)));

Why not NS_ReleaseOnMainThread(do_AddRef(this))? Or if this is not the last reference (probably not), you could use two NS_ReleaseOnMainThread()'s for the two fields, but that is extra work/allocations.

Also, instead of a new class you could add MediaEngineTabVideoSource::ClearMainThreadRefs() instead of DestroyRunnable::Run, and NewRunnableMethod/WrapRunnable.  That would be slightly less code.

Comment 39

[Andreas Pehrson [:pehrsons] (PTO; back in Aug)]

(In reply to Randell Jesup [:jesup] from comment #38)
> Comment on attachment 8786373 [details]
> Bug 1295352 - Release MediaEngineTabVideoSource main-thread-only members on
> main thread.
> 
> https://reviewboard.mozilla.org/r/75308/#review73560
> 
> Take your pick how to handle it.  Add a comment that those are
> main-thread-only and thus the Dispatch (or rename to DestroyMainthreadRefs
> or some such)
> 
> ::: dom/media/webrtc/MediaEngineTabVideoSource.cpp:207
> (Diff revision 1)
> >  
> >  nsresult
> >  MediaEngineTabVideoSource::Deallocate(AllocationHandle* aHandle)
> >  {
> >    MOZ_ASSERT(!aHandle);
> > +  NS_DispatchToMainThread(do_AddRef(new DestroyRunnable(this)));
> 
> Why not NS_ReleaseOnMainThread(do_AddRef(this))? Or if this is not the last
> reference (probably not), you could use two NS_ReleaseOnMainThread()'s for
> the two fields, but that is extra work/allocations.
> 
> Also, instead of a new class you could add
> MediaEngineTabVideoSource::ClearMainThreadRefs() instead of
> DestroyRunnable::Run, and NewRunnableMethod/WrapRunnable.  That would be
> slightly less code.

I thought that fitting in among existing code was more worth than saving a couple of lines.

I've added a comment to the DestroyRunnable declaration and to the declarations of the two members in question.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f2eecd2e2d6d
Check that there is no active gUM after each runTest test. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/6e5f577382c9
Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/3a4afeb8a046
Fix media mochitests to not end with active gUM. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/2bfe422ebb45
Check state in MediaEngines' NotifyPull(). r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/c809cf08cc13
Use mState in MediaEngineTabVideoSource. r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/6ee63db87f1e
Update mediaElementCapture tests to stop all tracks before finishing. r=jib
https://hg.mozilla.org/integration/mozilla-inbound/rev/a8c6adebc4ea
Release MediaEngineTabVideoSource main-thread-only members on main thread. r=jesup

Comment 41

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The Linux bc4 failure we talked about earlier on IRC started hitting, so I had to back this out for browser/base/content/test/webrtc/browser_devices_get_user_media.js | expected notification recording-device-events 

https://hg.mozilla.org/integration/mozilla-inbound/rev/20bc490e8cac0243ec214776708963510e4d2d90
https://hg.mozilla.org/integration/mozilla-inbound/rev/c014d7dc47943d9c1e38c1d90406be4a0602ec7f
https://hg.mozilla.org/integration/mozilla-inbound/rev/63d043a689cac950bcebd9eec0a7e583d2612f03
https://hg.mozilla.org/integration/mozilla-inbound/rev/ac2c86f6827e65a2699a35aef820dd7b193de500
https://hg.mozilla.org/integration/mozilla-inbound/rev/b527ef0a9dd25ab49a5a6cc5c977c10cc9880ea5
https://hg.mozilla.org/integration/mozilla-inbound/rev/e885b9705e2cbf44f0de657c6b54d7704f75c0fc
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d9a0aa9dff3f820ccc4b89fa3c4457587c8dbbb
https://hg.mozilla.org/integration/mozilla-inbound/rev/439f4fa25eed43bf36025d69862a3c445538fbae
https://hg.mozilla.org/integration/mozilla-inbound/rev/cb3960d28aa640a85692962c9b8265f55b5f2241
https://hg.mozilla.org/integration/mozilla-inbound/rev/8169af8a92061ce43eed24da499aee0a4c41e024
https://hg.mozilla.org/integration/mozilla-inbound/rev/63390c8490fa24416bdf1f2821feb805e25913f7
https://hg.mozilla.org/integration/mozilla-inbound/rev/e68e4ed5ab6e6692c90284443a3ad811aa35f183

Earliest Tier 1 failure: https://treeherder.mozilla.org/logviewer.html#?job_id=35034803&repo=mozilla-inbound

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/19d4e98a1191
Check that there is no active gUM after each runTest test. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f5c09304792
Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/d921975370c7
Fix media mochitests to not end with active gUM. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/081401df2032
Check state in MediaEngines' NotifyPull(). r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/9a2ec23ccac7
Use mState in MediaEngineTabVideoSource. r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/95f04e2c2b06
Update mediaElementCapture tests to stop all tracks before finishing. r=jib
https://hg.mozilla.org/integration/mozilla-inbound/rev/d0d0f847d043
Release MediaEngineTabVideoSource main-thread-only members on main thread. r=jesup

Comment 43

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/19d4e98a1191
https://hg.mozilla.org/mozilla-central/rev/4f5c09304792
https://hg.mozilla.org/mozilla-central/rev/d921975370c7
https://hg.mozilla.org/mozilla-central/rev/081401df2032
https://hg.mozilla.org/mozilla-central/rev/9a2ec23ccac7
https://hg.mozilla.org/mozilla-central/rev/95f04e2c2b06
https://hg.mozilla.org/mozilla-central/rev/d0d0f847d043

Comment 44

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/be887787de62
Backed out changeset d0d0f847d043 
https://hg.mozilla.org/integration/mozilla-inbound/rev/051a29aa9419
Backed out changeset 95f04e2c2b06 
https://hg.mozilla.org/integration/mozilla-inbound/rev/60e3863206b8
Backed out changeset 9a2ec23ccac7 
https://hg.mozilla.org/integration/mozilla-inbound/rev/aee84ca17a02
Backed out changeset 081401df2032 
https://hg.mozilla.org/integration/mozilla-inbound/rev/eecba1a05d03
Backed out changeset d921975370c7 
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d2f0fcf7185
Backed out changeset 4f5c09304792 
https://hg.mozilla.org/integration/mozilla-inbound/rev/4693d5445f67
Backed out changeset 19d4e98a1191 for frequently failing mda test test_getUserMedia_trackEnded.html on Linux x64 debug. r=backout

Comment 45

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry, backed bug 1295352 and bug 1294605 out for frequently failing mda test test_getUserMedia_trackEnded.html on Linux x64 debug:

Bug 1295352:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4693d5445f6755dea29f763a5568c456cbf3d8cb
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d2f0fcf7185290db6623557c66959e32b552c59
https://hg.mozilla.org/integration/mozilla-inbound/rev/eecba1a05d03f27cb3966948effe04a1c9bfec83
https://hg.mozilla.org/integration/mozilla-inbound/rev/aee84ca17a022f97e4544bd07e6dc54690020640
https://hg.mozilla.org/integration/mozilla-inbound/rev/60e3863206b89848720257e56b0a51fc454ec225
https://hg.mozilla.org/integration/mozilla-inbound/rev/051a29aa9419a32d6295f9e79362ceb3611b53a5
https://hg.mozilla.org/integration/mozilla-inbound/rev/be887787de62889d8761e4402b320f351d571d23

Bug 1294605:
https://hg.mozilla.org/integration/mozilla-inbound/rev/45b00bb6727a744bdc733578297418df6c9b1b58
https://hg.mozilla.org/integration/mozilla-inbound/rev/5a542246049e4ff7fc0c49234fdb49605047a8e2
https://hg.mozilla.org/integration/mozilla-inbound/rev/387012235302ccb073180d17ecea6e0d794dfcb3
https://hg.mozilla.org/integration/mozilla-inbound/rev/b82347df2602f07c7b190f0fbf7a6991d6022044
https://hg.mozilla.org/integration/mozilla-inbound/rev/40fb588cb1bc706a9c86dd5a27442714bd236afb

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=37972477&repo=mozilla-inbound

[task 2016-10-20T12:34:21.266367Z] 12:34:21     INFO -  2086 INFO TEST-PASS | dom/media/tests/mochitest/test_getUserMedia_trackEnded.html | Non-ended track should have readyState 'live'
[task 2016-10-20T12:34:21.267194Z] 12:34:21     INFO -  2087 INFO TEST-PASS | dom/media/tests/mochitest/test_getUserMedia_trackEnded.html | Non-ended track should have readyState 'live'
[task 2016-10-20T12:34:21.268021Z] 12:34:21     INFO -  2088 INFO iframe has been reset. Waiting for tracks to end.
[task 2016-10-20T12:34:21.268976Z] 12:34:21     INFO -  2089 INFO TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/test_getUserMedia_trackEnded.html | ended event never raised for track {27e92ba5-bd15-4636-bd27-50b7d180feba}
[task 2016-10-20T12:34:21.269789Z] 12:34:21     INFO -      allTracksEnded</<@dom/media/tests/mochitest/test_getUserMedia_trackEnded.html:44:25
[task 2016-10-20T12:34:21.270735Z] 12:34:21     INFO -      Async*allTracksEnded<@dom/media/tests/mochitest/test_getUserMedia_trackEnded.html:39:18
[task 2016-10-20T12:34:21.271640Z] 12:34:21     INFO -      @dom/media/tests/mochitest/test_getUserMedia_trackEnded.html:37:42
[task 2016-10-20T12:34:21.272537Z] 12:34:21     INFO -      Async*@dom/media/tests/mochitest/test_getUserMedia_trackEnded.html:28:12
[task 2016-10-20T12:34:21.273341Z] 12:34:21     INFO -      runTestWhenReady/<@dom/media/tests/mochitest/head.js:359:41
[task 2016-10-20T12:34:21.274221Z] 12:34:21     INFO -      Async*runTestWhenReady@dom/media/tests/mochitest/head.js:359:10
[task 2016-10-20T12:34:21.275018Z] 12:34:21     INFO -      runTest/<@dom/media/tests/mochitest/mediaStreamPlayback.js:274:15
[task 2016-10-20T12:34:21.275898Z] 12:34:21     INFO -      Async*runTest@dom/media/tests/mochitest/mediaStreamPlayback.js:273:31
[task 2016-10-20T12:34:21.276656Z] 12:34:21     INFO -      @dom/media/tests/mochitest/test_getUserMedia_trackEnded.html:22:3

Comment 46

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/c162c2edcb17
Backed out changeset d0d0f847d043 
https://hg.mozilla.org/mozilla-central/rev/3dcb945d820a
Backed out changeset 95f04e2c2b06 
https://hg.mozilla.org/mozilla-central/rev/07889c2aef31
Backed out changeset 9a2ec23ccac7 
https://hg.mozilla.org/mozilla-central/rev/8886d3fad3ab
Backed out changeset 081401df2032 
https://hg.mozilla.org/mozilla-central/rev/1c1c747944f2
Backed out changeset d921975370c7 
https://hg.mozilla.org/mozilla-central/rev/6baa5acba3fc
Backed out changeset 4f5c09304792 
https://hg.mozilla.org/mozilla-central/rev/f0f1aaf051d6
Backed out changeset 19d4e98a1191 for frequently failing mda test test_getUserMedia_trackEnded.html on Linux x64 debug. r=backout a=backout

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c827628cdf72
Check that there is no active gUM after each runTest test. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/f5e778110195
Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/572f17fd9e60
Fix media mochitests to not end with active gUM. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7c8bab2624a
Check state in MediaEngines' NotifyPull(). r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/f3dae14842ec
Use mState in MediaEngineTabVideoSource. r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/907d83f36492
Update mediaElementCapture tests to stop all tracks before finishing. r=jib
https://hg.mozilla.org/integration/mozilla-inbound/rev/57f7b75e5a62
Release MediaEngineTabVideoSource main-thread-only members on main thread. r=jesup

Comment 48

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c827628cdf72
https://hg.mozilla.org/mozilla-central/rev/f5e778110195
https://hg.mozilla.org/mozilla-central/rev/572f17fd9e60
https://hg.mozilla.org/mozilla-central/rev/a7c8bab2624a
https://hg.mozilla.org/mozilla-central/rev/f3dae14842ec
https://hg.mozilla.org/mozilla-central/rev/907d83f36492
https://hg.mozilla.org/mozilla-central/rev/57f7b75e5a62

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/24e39ddd4314
Check that there is no active gUM after each runTest test. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/16fc27df7bc0
Changed default MediaStreamPlayback.playMedia() to stop tracks afterwards. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/c72172d485b3
Fix media mochitests to not end with active gUM. r=pehrsons
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f06424634ea
Check state in MediaEngines' NotifyPull(). r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/ef7b7be0c4e8
Use mState in MediaEngineTabVideoSource. r=jesup
https://hg.mozilla.org/integration/mozilla-inbound/rev/6779dd0e2d28
Update mediaElementCapture tests to stop all tracks before finishing. r=jib
https://hg.mozilla.org/integration/mozilla-inbound/rev/6e16ba93d1d7
Release MediaEngineTabVideoSource main-thread-only members on main thread. r=jesup

Comment 50

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/24e39ddd4314
https://hg.mozilla.org/mozilla-central/rev/16fc27df7bc0
https://hg.mozilla.org/mozilla-central/rev/c72172d485b3
https://hg.mozilla.org/mozilla-central/rev/4f06424634ea
https://hg.mozilla.org/mozilla-central/rev/ef7b7be0c4e8
https://hg.mozilla.org/mozilla-central/rev/6779dd0e2d28
https://hg.mozilla.org/mozilla-central/rev/6e16ba93d1d7