1282897 - Run WebRTC tests on Autophone nexus-6p devices

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Dan Minor [:dminor]]

This is a tracking bug to get the WebRTC (Mw) job green on nexus-6p devices.

Comment 2

[Dan Minor [:dminor]]

Attachment: autophone.patch

Here's an updated manifest containing the tests we'd currently have to disable if we wanted to run on the nexus 6p devices. With the exception of test_getUserMedia_peerIdentity.html, all of these reproduce if I run the mochitests directly on my nexus 6p device.

Rather than file a bunch of bugs right away, I'm going to investigate these further and file bugs as I go.

Comment 3

[Dan Minor [:dminor]]

After disabling tests that are also disabled on the emulators, I'm left with the following list:

test_getUserMedia_addTrackRemoveTrack.html
test_getUserMedia_peerIdentity.html
test_getUserMedia_playAudioTwice.html
test_getUserMedia_playVideoAudioTwice.html
test_getUserMedia_playVideoTwice.html

Comment 4

[Dan Minor [:dminor]]

The test_getUserMedia_peerIdentity.html failure appears to be this intermittent:
https://bugzilla.mozilla.org/show_bug.cgi?id=1155666

Comment 5

[Dan Minor [:dminor]]

Attachment: Update manifest.

This disables tests that are currently disabled on the emulators as well as the four tests mentioned above that do currently run on the emulator. The intention is to start running the job on inbound trees which concurrently fixing the tests.

Comment 6

[Dan Minor [:dminor]]

Bob, assuming the manifest changes look good, could you please schedule the Mw job on mozilla-inbound and autoland? I've kicked off a try job here:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41accb6737db413444a1852c2433320d4500ce7

If test_getUserMedia_peerIdentity.html ends up being a high frequency intermittent, I think we should just disable it as well.

Comment 7

[Dan Minor [:dminor]]

The error I see is:

Ran 23 tests (1 parents, 22 subtests)
Expected results: 22
Unexpected results: 2 (ERROR: 1, FAIL: 1)

Unexpected Results
==================

dom/media/tests/mochitest/test_getUserMedia_playAudioTwice.html
---------------------------------------------------------------
FAIL Error executing test: Error: verifyPlaying timed out timeout/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:471:63 ...

Comment 8

[Dan Minor [:dminor]]

Attachment: Logcat captured from test_getUserMedia_playAudioTwice.html run

This is a logcat capture from a test_getUserMedia_playAudioTwice.html run. I've added some additional logging, prefixed by ">>>>".

Comment 9

[Dan Minor [:dminor]]

One difference between the emulators and the phone is that it does not appear that the MediaControlService is working properly on the emulators:

10-12 07:05:50.734   774   774 E dalvikvm: Could not find class 'android.app.Notification$Action$Builder', referenced from method org.mozilla.gecko.media.MediaControlService.createNotificationAction
10-12 07:05:50.734   774   774 W dalvikvm: VFY: unable to resolve new-instance 53 (Landroid/app/Notification$Action$Builder;) in Lorg/mozilla/gecko/media/MediaControlService;
10-12 07:05:50.744   774   774 D dalvikvm: VFY: replacing opcode 0x22 at 0x002b
10-12 07:05:50.744   774   856 D DLCStudyAction: Studying catalog..
10-12 07:05:50.754   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.754   774   856 V GeckoDLCCatalog: Waiting for catalog to be loaded
10-12 07:05:50.754   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.754   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x0048
10-12 07:05:50.754   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.754   774   857 D GeckoDLCCatalog: Loading from disk
10-12 07:05:50.764   774   857 D GeckoDLCCatalog: Catalog file does not exist: Bootstrapping initial catalog
10-12 07:05:50.774   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.784   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x009a
10-12 07:05:50.784   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.784   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.784   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x00a5

Comment 10

[Dan Minor [:dminor]]

Definitely the MediaControlService, removing the calls to stop at [1] and just below will cause the four tests to pass.

[1] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/media/MediaControlService.java#107

Comment 11

[Bob Clary [:bc] (inactive)]

Comment on attachment 8800218 [details] [diff] [review]
Update manifest.

Review of attachment 8800218 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f9bcd8d3d84a
Disable failing WebRTC Autophone tests on Android 6; r=bc

Comment 13

[Dan Minor [:dminor]]

Attachment: Add check for isAudioPlaying

This is sufficient for the tests to pass [1], but I'm not sure if it will cause problems elsewhere.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0988b6fbed35272312433f34784757c54259438d

Comment 14

[Alastor Wu [:alwu]]

Hi, Dan,

Although that fix can pass the test, I think that doesn't fix the real problem, because the tab.isAudioPlaying() should not be true when tab.isMediaPlaying() is false.

Could you briefly describe the situation how the test fail? Then we can figure out what's the real root cause.

Thanks!

Comment 15

[Andreas Pehrson [:pehrsons]]

I put this on IRC.

First see comment 8 for the problem.

[09:54:30] <pehrsons> The test is quite simple but you'll see in the log that when we run playMedia() the second time we become playing and then immediately paused
[09:54:34] <pehrsons> then we time out
[09:55:03] <pehrsons> in the test that's here: http://searchfox.org/mozilla-central/rev/cd1be634c9309c7fc99a3fde67dd44d343875f60/dom/media/tests/mochitest/test_getUserMedia_playAudioTwice.html#19

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f9bcd8d3d84a

Comment 17

[Dan Minor [:dminor]]

Attachment: Extra logcat logging from MediaControlService.java and Tabs.java

Attached are some extra logs in MediaControlService.java around lines 102 - 108 and in Tabs.java around lines 567 - 572 in case they are helpful.

Comment 18

[Alastor Wu [:alwu]]

Attachment: webrtc_debug.patch

Hi, Dan,
Could you help me check whether this patch can solve the fail?
If it can pass the test, I'll file a new bug and submit a patch.
Thanks!

Comment 19

[Dan Minor [:dminor]]

(In reply to Alastor Wu [:alwu] from comment #18)
> Created attachment 8801637 [details] [diff] [review]
> webrtc_debug.patch
> 
> Hi, Dan,
> Could you help me check whether this patch can solve the fail?
> If it can pass the test, I'll file a new bug and submit a patch.
> Thanks!

Try run is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b96455b35b95

Thanks!

Comment 20

[Bob Clary [:bc] (inactive)]

Please note that Autophone is having issues due to the change in the wifi infrastructure in the Mountain View office. As a result the try run this morning is unreliable. Autophone has been shut down until further notice.

Comment 21

[Dan Minor [:dminor]]

Alastor, I tested your changes on my local phone and it fixed failing tests. Unless there's a reason not to, I think it makes sense to land your patch while we wait for Autophone to come back online for final testing. Thanks for looking into this!

Comment 22

[Alastor Wu [:alwu]]

I'll solve this test fail in bug 1311245 :)

Comment 23

[Dan Minor [:dminor]]

(In reply to Bob Clary [:bc:] from comment #20)
> Please note that Autophone is having issues due to the change in the wifi
> infrastructure in the Mountain View office. As a result the try run this
> morning is unreliable. Autophone has been shut down until further notice.

I've retriggered the jobs just now and they are green :)

Comment 24

[Alastor Wu [:alwu]]

Comment on attachment 8800404 [details] [diff] [review]
Add check for isAudioPlaying

It would be done in bug1311245.

Comment 25

[Dan Minor [:dminor]]

Attachment: Update manifests to re-enable tests.

With Alastor's patch landed, we can re-enable these tests.

Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=14545ab7e915b24d39f3dacd2d34a544fb246a5c

Comment 26

[Bob Clary [:bc] (inactive)]

dminor: Is this sufficient coverage?

> test=runtestsremote.py Mw           repo=autoland-debug            devices=['nexus-6p-9']
> test=runtestsremote.py Mw           repo=autoland-opt              devices=['nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-central-debug     devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-central-opt       devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-inbound-debug     devices=['nexus-6p-7']
> test=runtestsremote.py Mw           repo=mozilla-inbound-opt       devices=['nexus-6p-7']
> test=runtestsremote.py Mw           repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py Mw           repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py R            repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py R            repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']

Once we merge, you would want this running on Aurora as well?

Comment 27

[Dan Minor [:dminor]]

(In reply to Bob Clary [:bc:] from comment #26)
> dminor: Is this sufficient coverage?
> 
> > test=runtestsremote.py Mw           repo=autoland-debug            devices=['nexus-6p-9']
> > test=runtestsremote.py Mw           repo=autoland-opt              devices=['nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-central-debug     devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-central-opt       devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-inbound-debug     devices=['nexus-6p-7']
> > test=runtestsremote.py Mw           repo=mozilla-inbound-opt       devices=['nexus-6p-7']
> > test=runtestsremote.py Mw           repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py Mw           repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py R            repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py R            repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> 
> Once we merge, you would want this running on Aurora as well?

Please drop the nexus-5 devices from central and try. I'd prefer to focus on getting the tests running properly on newer phones rather than worrying about the older ones. Other than that, looks good. Thanks!

You are right, we'll want Aurora once merged, assuming everything is looking ok.

Comment 28

[Bob Clary [:bc] (inactive)]

Comment on attachment 8806030 [details] [diff] [review]
Update manifests to re-enable tests.

r+

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d457a5086a6f
Update autophone webrtc manifests; r=bc

Comment 30

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d457a5086a6f

Comment 31

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.