1042795 - Extend WebRTC doorhangers test coverage

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Gian-Carlo Pascutto [:gcp]]

Bug 1018928 (tried to) extend the WebRTC UI tests greatly, but for some strange reason it was intermittent orange on try. Find out what some reason is and reland it.

Comment 1

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=ce5a44f09bad

Comment 2

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=349f5f040d7d

Comment 3

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=cc08852ee28e

Comment 4

[Gian-Carlo Pascutto [:gcp]]

Attachment: Expand Android guM + gUM doorhanger tests. r=

Expand tests for gUM, incorporate new constraints, adapt to new testing "HW".

Comment 5

[Gian-Carlo Pascutto [:gcp]]

Comment on attachment 8508774 [details] [diff] [review]
Expand Android guM + gUM doorhanger tests. r=

f? for jesup regarding media.getusermedia.screensharing.allowed_domains changes. I think I also need this in RELEASE_BUILD if we don't want try to go red on Beta/Release. example.com looks like it's IANA-owned, so this should be safe...I hope.

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 8508774 [details] [diff] [review]
Expand Android guM + gUM doorhanger tests. r=

>diff --git a/modules/libpref/init/all.js b/modules/libpref/init/all.js

> #ifdef RELEASE_BUILD
> pref("media.getusermedia.screensharing.allowed_domains", "webex.com,*.webex.com,collaborate.com,*.collaborate.com");
> #else
>  // temporary value, not intended for release - bug 1049087
>-pref("media.getusermedia.screensharing.allowed_domains", "mozilla.github.io,webex.com,*.webex.com,collaborate.com,*.collaborate.com");
>+pref("media.getusermedia.screensharing.allowed_domains", "mozilla.github.io,webex.com,*.webex.com,collaborate.com,*.collaborate.com,example.com");
> #endif

Does this mean the test will fail on any TBPL Release runs? Or do we disable this test on Release?

Comment 7

[Gian-Carlo Pascutto [:gcp]]

(In reply to Mark Finkle (:mfinkle) from comment #6)
> Does this mean the test will fail on any TBPL Release runs? Or do we disable
> this test on Release?

Based on jesup's f+ I will whitelist example.com in the release version.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/da0bf52f00c6

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Backed out for introducing new failures, i.e bug 1087348 and also the link below.
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e66efe38073

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3210239&repo=mozilla-inbound

I'll also point out that testGetUserMedia already has a number of other oranges filed on it, so this test doesn't have a very sterling reliability record as it is.

Comment 10

[Gian-Carlo Pascutto [:gcp]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #9)
> this test doesn't have a very sterling reliability
> record as it is.

Blaming the messenger :-)

06:54:05     INFO -  waitForText timeout on Would you like to share your camera and microphone with
06:54:05     INFO -  0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: Text string: 'Choose a tab to stream' is not found!

implies the gUM permissions dialog simply never appearing. I don't have a clue why that randomly wouldn't work, though.

https://bugzilla.mozilla.org/show_bug.cgi?id=1087348
05:30:00 INFO - waitForCondition timeout after 15000 ms.
05:30:00 WARNING - TEST-UNEXPECTED-FAIL | testGetUserMedia | Page title is correct - got gUM Test Page, expected video gumtest 

implies we set up the gUM dialog but the page never got a gUM success callback. Same question marks.

Comment 11

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=ed309ed10283

Comment 12

[Gian-Carlo Pascutto [:gcp]]

Android 2.3
01:01:44     INFO -  TEST-OK | testGetUserMedia | took 118359ms
01:01:15     INFO -  TEST-OK | testGetUserMedia | took 115335ms

vs

Android 4.0
10:48:05     INFO -  TEST-OK | testGetUserMedia | took 36114ms
00:57:30     INFO -  TEST-OK | testGetUserMedia | took 35394ms

Despite doing twice as much tests, Android 4.0 finishes more than 3 times faster.

Comment 13

[Gian-Carlo Pascutto [:gcp]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #12)
> Despite doing twice as much tests, Android 4.0 finishes more than 3 times
> faster.

That's just emulator vs real device of course.

Comment 14

[Gian-Carlo Pascutto [:gcp]]

Attachment: faillog.txt

Reproduced under the emulator.

Comment 15

[Gian-Carlo Pascutto [:gcp]]

E/Camera  (  432): Invalid range list string=null
E/WEBRTC-JC(  432): Camera doesn't know its own framerate, guessing 25fps.
D/WEBRTC-JC(  432): Camera 0, Facing back, Orientation 90
D/GeckoThumbnailHelper(  432): Using new thumbnail size: 242544 (width 326)
D/GeckoThumbnailHelper(  432): Sending thumbnail event: 326, 186
I/Gecko   (  432): void mozilla::AndroidBridge::HandleGeckoMessage(JSContext*, JS::HandleObject)
I/Gecko   (  432): void mozilla::AndroidBridge::HandleGeckoMessage(JSContext*, JS::HandleObject)
I/dalvikvm(  432): Could not find method android.widget.Spinner.<init>, referenced from method org.mozilla.gecko.prompts.PromptInput$MenulistInput.getView
W/dalvikvm(  432): VFY: unable to resolve direct method 12029: Landroid/widget/Spinner;.<init> (Landroid/content/Context;I)V
D/dalvikvm(  432): VFY: replacing opcode 0x70 at 0x005a
D/dalvikvm(  432): VFY: dead code 0x005d-005f in Lorg/mozilla/gecko/prompts/PromptInput$MenulistInput;.getView (Landroid/content/Context;)Landroid/view/View;

The above is interesting. You can see the camera enumeration, and I think the VFY warnings are actually from the code that makes the list of cameras for the Doorhanger, so all the right code is being called.

Now, the next thing in the log is a bunch of goo from SUTAgent? And it's complaining it can't connect the input? Geoff, does this ring a bell?

D/        (   64): HostConnection::get() New Host Connection established 0x350e30, tid 265
I/Watcher (  293): Starting SUTAgent from watcher code
I/ActivityManager(   64): Starting: Intent { act=android.intent.action.MAIN flg=0x10000000 pkg=com.mozilla.SUTAgentAndroid cmp=com.mozilla.SUTAgentAndroid/.SUTAgentAndroid } from pid 293
D/GeckoHealthRec(  432): Recording session end: P
V/GeckoHealthRec(  432): Recorded session entry for env 1, current is 1
D/GeckoSessInfo(  432): Recording session done: 1414429331551
I/ActivityManager(   64): Start proc com.mozilla.SUTAgentAndroid for activity com.mozilla.SUTAgentAndroid/.SUTAgentAndroid: 
I/SUTAgentAndroid(  500): SUTAgentAndroid Version 1.20
E/gralloc_goldfish(  432): gralloc_unregister_buffer(0x2a2000): invalid buffer
W/GraphicBufferMapper(  432): unregisterBuffer(0x2a2000) failed -22 (Invalid argument)
D/dalvikvm(  500): GC_EXPLICIT freed 64K, 50% free 2736K/5379K, external 716K/1038K, paused 61ms
I/SUTAgentAndroid(  500): Changed permissions on /data/local/tmp to make it writable: Changing permissions for /data/local/tmp
I/SUTAgentAndroid(  500):       <empty>
I/SUTAgentAndroid(  500):       chmod /data/local/tmp ok
I/SUTAgentAndroid(  500): onCreate
I/SUTAgentAndroid(  500): Set device root to /mnt/sdcard
I/SUTAgentAndroid(  500): Test Root: /mnt/sdcard
D/        (  500): HostConnection::get() New Host Connection established 0x1f56a8, tid 500
D/        (   64): HostConnection::get() New Host Connection established 0x257880, tid 267
I/ActivityManager(   64): Displayed com.mozilla.SUTAgentAndroid/.SUTAgentAndroid: +868ms
I/Gecko   (  432): void mozilla::AndroidBridge::HandleGeckoMessage(JSContext*, JS::HandleObject)
I/Robocop (  432): {"message":"true should equal true","time":1414429363397,"source":"robocop","status":"PASS","test":"testGetUserMedia","thread":null,"subtest":"getUserMedia found a camera","action":"test_status","pid":null}
I/Robocop (  432): {"message":"true should equal true","time":1414429364409,"source":"robocop","status":"PASS","test":"testGetUserMedia","thread":null,"subtest":"getUserMedia doorhanger has been displayed","action":"test_status","pid":null}
W/InputDispatcher(   64): Permission denied: injecting event from pid 432 uid 10017 to window with input channel 40720948 com.mozilla.SUTAgentAndroid/com.mozilla.SUTAgentAndroid.SUTAgentAndroid (server) owned by uid 10032
W/WindowManager(   64): Input event injection permission denied.

Comment 16

[Geoff Brown [:gbrown]]

I have seen this sort of thing before and I'm pretty sure what's happening is that sutagent is being killed on memory pressure before this log starts. Then sutagent restarts and goes to foreground during your robocop test and then some robocop operations (probably a click) start to fail: "Input event injection permission denied" is because the robocop click (or whatever) is being directed to the foreground app, sutagent, which robocop does not have permission for.

sutagent should live for the duration of the test...really it should start with the emulator startup and keep running until the emulator is shutdown. You need to figure out why it is being killed off.

Comment 17

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=ab7cbb9a65ff

Comment 18

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=8eee23b35ede

Comment 19

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=8dad5515d1d9

Comment 20

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=2558844f8fe3

Comment 21

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=55bfe8906eb7

Comment 22

[Gian-Carlo Pascutto [:gcp]]

https://tbpl.mozilla.org/?tree=Try&rev=2230ca8fffb0

Comment 23

[Gian-Carlo Pascutto [:gcp]]

Attachment: Expand Android guM + gUM doorhanger tests. r=

That last try is finally looking good. Changes from earlier version:
1) We stop the gUM streams immediately after verifying that they have
audio and video tracks. This stops the CPU of the test hardware from
being pegged.
2) For each and every UI interaction, wait for the UI element that
we want to click on to be actually there, and wait for it to disappear
after clicking before continuing again. Liberally sprinkle mAsserter.is
checks between those so we see exactly what goes wrong. I think this
eliminated some race conditions in the UI interaction that happened
on try.

Comment 24

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 8513630 [details] [diff] [review]
Expand Android guM + gUM doorhanger tests. r=

Something I wanted to point out, just in case it has some bearing in the future.

>diff --git a/mobile/android/base/tests/robocop_getusermedia.html b/mobile/android/base/tests/robocop_getusermedia.html

>   <script type="application/javascript">
>   var video_status = false;
>   var video = document.createElement("video");
>   video.setAttribute("width", 640);
>   video.setAttribute("height", 480);
> 
>   var audio_status = false;
>   var audio = document.createElement("audio");
>   audio.setAttribute("controls", true);
> 
>+  var content = document.getElementById("content");
>+  document.title = "gUM Test Page";

In the HTML files, you are touching the DOM in the global scope of the script. This can be dangerous, since the DOM might not be ready yet.

It is typical for web pages to wait for "DOMContentLoaded" or "load" before kicking off any DOM activity.

Just something to keep in mind, in case weird errors ever start happening.

Comment 25

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d7a6ecdac6b4

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/d7a6ecdac6b4