Closed Bug 1282897 Opened 4 years ago Closed 3 years ago

Run WebRTC tests on Autophone nexus-6p devices

Categories

(Testing :: General, defect, P1)

Version 3
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dminor, Assigned: dminor)

References

Details

Attachments

(6 files, 1 obsolete file)

This is a tracking bug to get the WebRTC (Mw) job green on nexus-6p devices.
Duplicate of this bug: 1294508
Depends on: 1294511
Attached patch autophone.patch (obsolete) — Splinter Review
Here's an updated manifest containing the tests we'd currently have to disable if we wanted to run on the nexus 6p devices. With the exception of test_getUserMedia_peerIdentity.html, all of these reproduce if I run the mochitests directly on my nexus 6p device.

Rather than file a bunch of bugs right away, I'm going to investigate these further and file bugs as I go.
Assignee: nobody → dminor
Priority: -- → P1
After disabling tests that are also disabled on the emulators, I'm left with the following list:

test_getUserMedia_addTrackRemoveTrack.html
test_getUserMedia_peerIdentity.html
test_getUserMedia_playAudioTwice.html
test_getUserMedia_playVideoAudioTwice.html
test_getUserMedia_playVideoTwice.html
The test_getUserMedia_peerIdentity.html failure appears to be this intermittent:
https://bugzilla.mozilla.org/show_bug.cgi?id=1155666
Attached patch Update manifest.Splinter Review
This disables tests that are currently disabled on the emulators as well as the four tests mentioned above that do currently run on the emulator. The intention is to start running the job on inbound trees which concurrently fixing the tests.
Attachment #8797262 - Attachment is obsolete: true
Attachment #8800218 - Flags: review?(bob)
Bob, assuming the manifest changes look good, could you please schedule the Mw job on mozilla-inbound and autoland? I've kicked off a try job here:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41accb6737db413444a1852c2433320d4500ce7

If test_getUserMedia_peerIdentity.html ends up being a high frequency intermittent, I think we should just disable it as well.
Flags: needinfo?(bob)
The error I see is:

Ran 23 tests (1 parents, 22 subtests)
Expected results: 22
Unexpected results: 2 (ERROR: 1, FAIL: 1)

Unexpected Results
==================

dom/media/tests/mochitest/test_getUserMedia_playAudioTwice.html
---------------------------------------------------------------
FAIL Error executing test: Error: verifyPlaying timed out timeout/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:471:63 ...
This is a logcat capture from a test_getUserMedia_playAudioTwice.html run. I've added some additional logging, prefixed by ">>>>".
One difference between the emulators and the phone is that it does not appear that the MediaControlService is working properly on the emulators:

10-12 07:05:50.734   774   774 E dalvikvm: Could not find class 'android.app.Notification$Action$Builder', referenced from method org.mozilla.gecko.media.MediaControlService.createNotificationAction
10-12 07:05:50.734   774   774 W dalvikvm: VFY: unable to resolve new-instance 53 (Landroid/app/Notification$Action$Builder;) in Lorg/mozilla/gecko/media/MediaControlService;
10-12 07:05:50.744   774   774 D dalvikvm: VFY: replacing opcode 0x22 at 0x002b
10-12 07:05:50.744   774   856 D DLCStudyAction: Studying catalog..
10-12 07:05:50.754   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.754   774   856 V GeckoDLCCatalog: Waiting for catalog to be loaded
10-12 07:05:50.754   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.754   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x0048
10-12 07:05:50.754   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.754   774   857 D GeckoDLCCatalog: Loading from disk
10-12 07:05:50.764   774   857 D GeckoDLCCatalog: Catalog file does not exist: Bootstrapping initial catalog
10-12 07:05:50.774   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.784   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x009a
10-12 07:05:50.784   774   774 I dalvikvm: Could not find method android.media.session.MediaController.getTransportControls, referenced from method org.mozilla.gecko.media.MediaControlService.handleIntent
10-12 07:05:50.784   774   774 W dalvikvm: VFY: unable to resolve virtual method 1358: Landroid/media/session/MediaController;.getTransportControls ()Landroid/media/session/MediaController$TransportControls;
10-12 07:05:50.784   774   774 D dalvikvm: VFY: replacing opcode 0x6e at 0x00a5
Definitely the MediaControlService, removing the calls to stop at [1] and just below will cause the four tests to pass.

[1] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/media/MediaControlService.java#107
Comment on attachment 8800218 [details] [diff] [review]
Update manifest.

Review of attachment 8800218 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm.
Attachment #8800218 - Flags: review?(bob) → review+
Pushed by dminor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f9bcd8d3d84a
Disable failing WebRTC Autophone tests on Android 6; r=bc
This is sufficient for the tests to pass [1], but I'm not sure if it will cause problems elsewhere.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0988b6fbed35272312433f34784757c54259438d
Attachment #8800404 - Flags: feedback?(alwu)
Hi, Dan,

Although that fix can pass the test, I think that doesn't fix the real problem, because the tab.isAudioPlaying() should not be true when tab.isMediaPlaying() is false.

Could you briefly describe the situation how the test fail? Then we can figure out what's the real root cause.

Thanks!
Flags: needinfo?(dminor)
I put this on IRC.

First see comment 8 for the problem.

[09:54:30] <pehrsons> The test is quite simple but you'll see in the log that when we run playMedia() the second time we become playing and then immediately paused
[09:54:34] <pehrsons> then we time out
[09:55:03] <pehrsons> in the test that's here: http://searchfox.org/mozilla-central/rev/cd1be634c9309c7fc99a3fde67dd44d343875f60/dom/media/tests/mochitest/test_getUserMedia_playAudioTwice.html#19
Flags: needinfo?(dminor)
Attached are some extra logs in MediaControlService.java around lines 102 - 108 and in Tabs.java around lines 567 - 572 in case they are helpful.
Hi, Dan,
Could you help me check whether this patch can solve the fail?
If it can pass the test, I'll file a new bug and submit a patch.
Thanks!
Flags: needinfo?(alwu) → needinfo?(dminor)
(In reply to Alastor Wu [:alwu] from comment #18)
> Created attachment 8801637 [details] [diff] [review]
> webrtc_debug.patch
> 
> Hi, Dan,
> Could you help me check whether this patch can solve the fail?
> If it can pass the test, I'll file a new bug and submit a patch.
> Thanks!

Try run is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b96455b35b95

Thanks!
Flags: needinfo?(dminor)
Please note that Autophone is having issues due to the change in the wifi infrastructure in the Mountain View office. As a result the try run this morning is unreliable. Autophone has been shut down until further notice.
Alastor, I tested your changes on my local phone and it fixed failing tests. Unless there's a reason not to, I think it makes sense to land your patch while we wait for Autophone to come back online for final testing. Thanks for looking into this!
Flags: needinfo?(alwu)
Depends on: 1311245
I'll solve this test fail in bug 1311245 :)
Flags: needinfo?(alwu)
(In reply to Bob Clary [:bc:] from comment #20)
> Please note that Autophone is having issues due to the change in the wifi
> infrastructure in the Mountain View office. As a result the try run this
> morning is unreliable. Autophone has been shut down until further notice.

I've retriggered the jobs just now and they are green :)
Comment on attachment 8800404 [details] [diff] [review]
Add check for isAudioPlaying

It would be done in bug1311245.
Attachment #8800404 - Flags: feedback?(alwu)
With Alastor's patch landed, we can re-enable these tests.

Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=14545ab7e915b24d39f3dacd2d34a544fb246a5c
Attachment #8806030 - Flags: review?(bob)
dminor: Is this sufficient coverage?

> test=runtestsremote.py Mw           repo=autoland-debug            devices=['nexus-6p-9']
> test=runtestsremote.py Mw           repo=autoland-opt              devices=['nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-central-debug     devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-central-opt       devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> test=runtestsremote.py Mw           repo=mozilla-inbound-debug     devices=['nexus-6p-7']
> test=runtestsremote.py Mw           repo=mozilla-inbound-opt       devices=['nexus-6p-7']
> test=runtestsremote.py Mw           repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py Mw           repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py R            repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> test=runtestsremote.py R            repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']

Once we merge, you would want this running on Aurora as well?
Flags: needinfo?(bob) → needinfo?(dminor)
(In reply to Bob Clary [:bc:] from comment #26)
> dminor: Is this sufficient coverage?
> 
> > test=runtestsremote.py Mw           repo=autoland-debug            devices=['nexus-6p-9']
> > test=runtestsremote.py Mw           repo=autoland-opt              devices=['nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-central-debug     devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-central-opt       devices=['nexus-4-7', 'nexus-5-5', 'nexus-6p-9']
> > test=runtestsremote.py Mw           repo=mozilla-inbound-debug     devices=['nexus-6p-7']
> > test=runtestsremote.py Mw           repo=mozilla-inbound-opt       devices=['nexus-6p-7']
> > test=runtestsremote.py Mw           repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py Mw           repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py R            repo=try-debug                 devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> > test=runtestsremote.py R            repo=try-opt                   devices=['nexus-4-7', 'nexus-5-6', 'nexus-6-2', 'nexus-6p-10', 'nexus-9-2']
> 
> Once we merge, you would want this running on Aurora as well?

Please drop the nexus-5 devices from central and try. I'd prefer to focus on getting the tests running properly on newer phones rather than worrying about the older ones. Other than that, looks good. Thanks!

You are right, we'll want Aurora once merged, assuming everything is looking ok.
Flags: needinfo?(dminor)
Comment on attachment 8806030 [details] [diff] [review]
Update manifests to re-enable tests.

r+
Attachment #8806030 - Flags: review?(bob) → review+
Pushed by dminor@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d457a5086a6f
Update autophone webrtc manifests; r=bc
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.