1472298 - add speedometer benchmark to raptor/android

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Comment 1

[Robert Wood [:rwood]]

We are seeing add-on errors when attempting to load the raptor webext with the geckoview example app. Even a slimmed down raptor webext shows the same issues.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: motog5-raptor-test-platforms.patch

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: motog5-raptor-test-sets.patch

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: motog5-raptor-test-platform-assignments.patch

Comment 5

[Bob Clary [:bc] (inactive)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e47b8ad301b6c311e2c7a18e59fc9b107fcf8fd4&filter-tier=1&filter-tier=2&filter-tier=3

fails using mozinstall to install geckoview_example.apk

Comment 6

[Robert Wood [:rwood]]

Comment on attachment 9006595 [details] [diff] [review]
motog5-raptor-test-platforms.patch

Review of attachment 9006595 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 7

[Robert Wood [:rwood]]

Comment on attachment 9006596 [details] [diff] [review]
motog5-raptor-test-sets.patch

Review of attachment 9006596 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 8

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cb759edcc70b41e9a09858ca994dba383fa87c4

Comment 9

[Robert Wood [:rwood]]

Comment on attachment 9006597 [details] [diff] [review]
motog5-raptor-test-platform-assignments.patch

Review of attachment 9006597 [details] [diff] [review]:
-----------------------------------------------------------------

Also seeing the same error on try (mozinstall failing to install the geckoview_example app).

::: testing/mozharness/configs/raptor/android_hw_config.py
@@ +1,2 @@
> +import os
> +#import platform

nit: remove commented out line

@@ +1,4 @@
> +import os
> +#import platform
> +
> +#VENV_PATH = '%s/build/venv' % os.getcwd()

nit: remove commented out line

::: testing/mozharness/mozharness/mozilla/testing/raptor.py
@@ +133,5 @@
>          self.test = None
>          self.gecko_profile = self.config.get('gecko_profile')
>          self.gecko_profile_interval = self.config.get('gecko_profile_interval')
> +        self.test_packages_url = self.config.get('test_packages_url')
> +        self.test_suite = self.config.get('test_suite')

Where is self.test_suite used (somewhere in the land of bitbar?)

Comment 10

[Bob Clary [:bc] (inactive)]

No where that I know. I added it to the last patch when I added the test_packages_url to fix the problem there. We can remove it. These patches+reviews were mostly to get the patches to you and gbrown so you could look at them and figure out the problems while I'm unavailable.

Comment 11

[Robert Wood [:rwood]]

Ok great thanks :bc!

Comment 12

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I'm running into some trouble around argument passing. In particular, I see:

https://dxr.mozilla.org/mozilla-central/rev/c2e3be6a1dd352b969a45f0b85e87674e24ad284/testing/mozharness/mozharness/mozilla/testing/raptor.py#396

    def run_tests(self, args=None, **kw):

but run_tests(self) is just a mozharness script "step" function, which, afaik, doesn't take parameters.

Regardless of options passed to the script, I find kw = {}.

On the other hand, I see a similar run_tests signature in Talos.

:rwood - Any insight?

Comment 13

[Geoff Brown [:gbrown] (pto Apr 11-18)]

In other news, I don't see any code to install geckoview after the apk is downloaded. I'm going to re-use the apk installation code from android_hardware_unittest.py, first refactoring it so that code can be shared between the various mozharness scripts for android.

Comment 14

[Geoff Brown [:gbrown] (pto Apr 11-18)]

This almost worked:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=72fa647736c48253baa668108411c48f90219801

Comment 15

[Bob Clary [:bc] (inactive)]

1. It would be nice to dump the logcat to see what the device is doing.
2. The control server appears to be working. How can we tell the mitmproxy is working? We can ask bitbar if there are any networking issues which might be a problem.
3. Perhaps this is motog5 related. I could move some p2 devices into the p2 perf device group and we could enable speedometer on them as well.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

we shouldn't have mitmproxy for speedometer, but we do have 2 different ports we serve data from (control server, data server).  I agree that the p2 devices would be worth trying

Comment 17

[Bob Clary [:bc] (inactive)]

Attachment: motog5-raptor-test-platform-assignments.patch

updated for rwood's comments.

Comment 18

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16)
> we shouldn't have mitmproxy for speedometer, but we do have 2 different
> ports we serve data from (control server, data server).  I agree that the p2
> devices would be worth trying

Yeah mitmproxy won't start unless you try to run tp6.

Comment 19

[Robert Wood [:rwood]]

(In reply to Geoff Brown [:gbrown] from comment #13)
> In other news, I don't see any code to install geckoview after the apk is
> downloaded. I'm going to re-use the apk installation code from
> android_hardware_unittest.py, first refactoring it so that code can be
> shared between the various mozharness scripts for android.

I'm assuming that uses MozDevice ADBAndroid.install_app? (I'll check)

Comment 20

[Robert Wood [:rwood]]

(In reply to Geoff Brown [:gbrown] from comment #12)
> I'm running into some trouble around argument passing. In particular, I see:
> 
> https://dxr.mozilla.org/mozilla-central/rev/
> c2e3be6a1dd352b969a45f0b85e87674e24ad284/testing/mozharness/mozharness/
> mozilla/testing/raptor.py#396
> 
>     def run_tests(self, args=None, **kw):
> 
> but run_tests(self) is just a mozharness script "step" function, which,
> afaik, doesn't take parameters.
> 
> Regardless of options passed to the script, I find kw = {}.
> 
> On the other hand, I see a similar run_tests signature in Talos.
> 
> :rwood - Any insight?

I'm a bit confused, how are you invoking Raptor? You shoudn't call run_tests directly...

Comment 21

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #19)
> (In reply to Geoff Brown [:gbrown] from comment #13)
> > In other news, I don't see any code to install geckoview after the apk is
> > downloaded. I'm going to re-use the apk installation code from
> > android_hardware_unittest.py, first refactoring it so that code can be
> > shared between the various mozharness scripts for android.
> 
> I'm assuming that uses MozDevice ADBAndroid.install_app? (I'll check)

Oh I see there's a mozharness android mixin cool

Comment 22

[Bob Clary [:bc] (inactive)]

Attachment: p2-raptor-test-platforms.patch

Not looking for a review really. These patches are to enable the p2 testing. We can deal with reviews when we are ready to go live.

Comment 23

[Bob Clary [:bc] (inactive)]

Attachment: p2-raptor-test-sets.patch

Comment 24

[Bob Clary [:bc] (inactive)]

Attachment: p2-raptor-test-platform-assignments.patch

Comment 25

[Bob Clary [:bc] (inactive)]

oops, mixed a couple of patches up. I'll fix up but applying them all will fix the issue.

Comment 26

[Bob Clary [:bc] (inactive)]

Attachment: p2-raptor-test-platforms.patch

Comment 27

[Bob Clary [:bc] (inactive)]

Attachment: p2-raptor-test-sets.patch

Comment 28

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #20)
> 
> I'm a bit confused, how are you invoking Raptor? You shoudn't call run_tests
> directly...

NVM I see you got it to work with:

01:08:44     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', u'/builds/worker/workspace/build/tests/raptor/raptor/raptor.py', u'--test', 'raptor-speedometer', u'--symbolsPath', 'https://queue.taskcluster.net/v1/task/A4EzNSaGSDOpjHKUrRMfHA/artifacts/public/build/target.crashreporter-symbols.zip', u'--app', 'geckoview', u'--branchName', 'try', u'--binary', 'org.mozilla.geckoview_example', u'--log-tbpl-level=debug'] with output_timeout 3600

Is there any way that we can run this and watch the device live to see what is going on? On your run in comment 14 looks like it is all starting but then it times out. Either it is really really slow and it is timing out before the test is done, or the test isn't starting for some reason... the adb reverse is there so it should be finding the served-out speedometer source.

Comment 29

[Bob Clary [:bc] (inactive)]

try run without the fixes for installing.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dec04f0407e405f000f9b54a9d0c9a8d8c934d94&filter-tier=1&filter-tier=2&filter-tier=3

p2 aarch64 does not work with --artifact. do this to exclude it

./mach try fuzzy --full --query "'test-android-hw arm7 raptor-speedometer-geckoview" --artifact

live log doesn't work unfortunately due to issues with taskcluster-worker. If you dump the logcat, you'll be able to see more about what the device is doing.

Comment 30

[Geoff Brown [:gbrown] (pto Apr 11-18)]

With logcat artifact:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f50647b61baffead6f91d093b317c3148c35782a

Comment 31

[Robert Wood [:rwood]]

Thanks :gbrown for adding the logcat artifact. Retriggered some. Speedometer is running on the device but is intermittently failing with:

09-06 07:37:00.601 14720 14735 E GeckoConsole: [JavaScript Error: "Error: Load timeout for modules: app/page/app,jquery
09-06 07:37:00.601 14720 14735 E GeckoConsole: http://requirejs.org/docs/errors.html#timeout" {file: "http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js" line: 168}]
09-06 07:37:00.601 14720 14735 E GeckoConsole: makeError@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:168:17
09-06 07:37:00.601 14720 14735 E GeckoConsole: checkLoaded@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:698:23
09-06 07:37:00.601 14720 14735 E GeckoConsole: checkLoaded/checkLoadedTimeoutId<@http://localhost:49235/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:720:25

On the run above in comment 30, you'll see the green run doesn't have that error. Nice to see it does work, but for some reason there's this intermittent.

Comment 32

[Robert Wood [:rwood]]

Comment on attachment 9006902 [details] [diff] [review]
p2-raptor-test-platform-assignments.patch

Review of attachment 9006902 [details] [diff] [review]:
-----------------------------------------------------------------

Great thanks

Comment 33

[Robert Wood [:rwood]]

Just adding :gbrown's latest try push:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=305337495d7d7a3ca65f9b04d161825eb88008ca

On one of the failures I noticed the weather app was doing an update (not sure if that caused speedometer to stop/timeout). The bitbar folks have since 'disabled Time and Weather system app from all Moto G5s' so perhaps that will help.

Comment 34

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: support apk install, logcat, etc

This works for me, with :bc's patches.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6a0f768436a746ba37d6506cf12d992664b1368

Comment 35

[Bob Clary [:bc] (inactive)]

I did a bunch of tests using Geoff's patches

<https://treeherder.mozilla.org/#/jobs?repo=try&revision=866fd6509fbd8f19af719e597ed7106ffaed3be2&filter-tier=1&filter-tier=2&filter-tier=3>

Interesting fact 1: p2 x86_64 is slower than x86.

Interesting fact 2: p2s are pretty reliable with no oranges out of 10 and seem to have pretty consistent values.

Interesting fact 3: motog5 failures are intermittent but too common. If we can't figure it out we could alleviate the problem by reducing the timeout and keeping them as tier 3.

This seems to be the issue

09-06 20:29:35.977  5012  5027 E GeckoConsole: [JavaScript Error: "Error: Load timeout for modules: app/page/app,jquery
09-06 20:29:35.977  5012  5027 E GeckoConsole: http://requirejs.org/docs/errors.html#timeout" {file: "http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js" line: 168}]
09-06 20:29:35.977  5012  5027 E GeckoConsole: makeError@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:168:17
09-06 20:29:35.977  5012  5027 E GeckoConsole: checkLoaded@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:698:23
09-06 20:29:35.977  5012  5027 E GeckoConsole: checkLoaded/checkLoadedTimeoutId<@http://localhost:38347/Speedometer/resources/todomvc/dependency-examples/flight/flight/node_modules/requirejs/require.js:720:25

Is that time out configurable?

Comment 36

[Joel Maher ( :jmaher ) (UTC -8)]

interesting observation and awesome try push :)

looking at this, I first thought we were hitting http://requirejs.org, but that is just an error message- we are really localhost.  Digging a bit deeper, I think we are timing out to load the jquery which is 320K in size- is it possible we are running out of memory on the g5?


I looked at a logcat artifact uploaded (which is really useful btw):
https://queue.taskcluster.net/v1/task/J7Hid377SKmdfTdLBUBRbQ/runs/0/artifacts/public/test_info/logcat-ZY322M554N.log

There are a handful of Javascript Errors, most don't seem to be related.  In fact we have a 20 minute timeout inside the browser, but our normal runtime is <10 minutes- I am not sure if extending the timeout will help or not.

Another thing I noticed is the screenshot isn't getting uploaded, small details, but that might have a clue.  I see a few messages in the logcat indicating that gecko is in the background, worth looking into to see if we can disable more services.

We could get speedometer on P2 running 'now' and sort out the G5 when we have more time (i.e. after getting Unity-webgl and battery tests up)

Comment 37

[Robert Wood [:rwood]]

I was able to reproduce the same intermittent locally on a Moto G5 that I borrowed - once via raptor and twice via running the speedometer (1.0) benchmark via wifi directly on the speedometer benchmark site in the geckoview_example app. There was lots of memory still available on the device even when the intermittent did happen.

After that I wanted to get extra info (i.e. adb shell dumpsys meminfo) and I couldn't repro the failure again. Tried for a couple of hours and couldn't repro it again - it now it seems to work great hah. I think we should go ahead and land the patches and then revisit this again later and see how it runs on the G5. I am hesitant to file a geckoview bug until we can reproduce it consistently.

Comment 38

[Robert Wood [:rwood]]

Ok, so are we ready to land? Do you guys want to land your patches separately but at the same time?

Comment 39

[Bob Clary [:bc] (inactive)]

Let's wait until we have everyone together on irc this afternoon and we can decide/coordinate.

Comment 40

[Bob Clary [:bc] (inactive)]

The patches as is need to land together as a group since gbrown's depends on the existence of testing/mozharness/configs/raptor/android_hw_config.py and we will get installation failures without gbrown's patch. Or we could refactor a couple and move the android_hw_config.py creation to gbrown's patch and land it first, followed by the others. Or we could just live with installation failures for one push if we landed gbrown's separately.

I would vote for gbrown landing everything but am open to other approaches.

Comment 41

[Bob Clary [:bc] (inactive)]

Comment on attachment 9006890 [details] [diff] [review]
motog5-raptor-test-platform-assignments.patch

gbrown already reviewed but the check in comment references both him and rwood.

Comment 42

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Comment on attachment 9007097 [details] [diff] [review]
support apk install, logcat, etc

Most of the android.py code is copied from android_hardware_unittest.py, or android_emulator_unittest.py. I plan to remove duplicate code from those scripts and have all the android mozharness scripts use android.py -- one day, in another bug.

Comment 43

[Bob Clary [:bc] (inactive)]

Comment on attachment 9007097 [details] [diff] [review]
support apk install, logcat, etc

Review of attachment 9007097 [details] [diff] [review]:
-----------------------------------------------------------------

r+. Looks ok to me. Couple of style issues that aren't important enough to call out.

I would really love for use to be able to use mozdevice, mozscreenshot and other stuff from mozbase in mozharness. Maybe we can think about that for the future.

::: testing/mozharness/mozharness/mozilla/testing/android.py
@@ +188,5 @@
> +                       (self.installer_path, self.device_name),
> +                       EXIT_STATUS_DICT[TBPL_RETRY])
> +        return install_ok
> +
> +    def screenshot(self, prefix):

This only works on the emulator. We added the capability in bug 1482878 to also do hardware using dump_device_screen.

Is there anyway for us to get mozscreenshot into mozharness? Maybe a followup?

::: testing/mozharness/mozharness/mozilla/testing/raptor.py
@@ +487,5 @@
>                                                  env=env)
> +
> +        if self.app == "geckoview":
> +            self.logcat_stop()
> +            self.screenshot("raptor-screenshot-end")

See comment in the AndroidMixin

Comment 44

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Bob Clary [:bc:] from comment #43)
> This only works on the emulator. We added the capability in bug 1482878 to
> also do hardware using dump_device_screen.

Thanks, I was wondering about that. I'll remove the screenshot calls from raptor, and we can follow up on device screenshot for raptor later.

> Is there anyway for us to get mozscreenshot into mozharness? Maybe a
> followup?

That is possible, but awkward (standard way is to duplicate the mozbase project in mozharness). Anyway, yes, let's consider that in a followup.

Comment 45

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Once we get these last few reviews done, I'll check one more time on try, then land all the patches...hopefully soon!

Comment 46

[Robert Wood [:rwood]]

Comment on attachment 9006902 [details] [diff] [review]
p2-raptor-test-platform-assignments.patch

Let's put the Moto G5 as tier 3 due to the intermittent, and the Google Pixel 2 as tier 2 since it seems to be stable. Thanks!

Comment 47

[Geoff Brown [:gbrown] (pto Apr 11-18)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fabf4b1512b0ba7d7ce2aabc83c3ec2a776226d5&filter-tier=1&filter-tier=2&filter-tier=3

Comment 48

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #46)
> Comment on attachment 9006902 [details] [diff] [review]
> p2-raptor-test-platform-assignments.patch
> 
> Let's put the Moto G5 as tier 3 due to the intermittent, and the Google
> Pixel 2 as tier 2 since it seems to be stable. Thanks!

Oh I see that the raptor.yml is also updated in the patch in comment 41 (and has the G5 tier 3, default tier 2 which is what we want).

Comment 49

[Bob Clary [:bc] (inactive)]

gbrown: want to go ahead and handle the tier change that rwood requested for p2-raptor-test-platform-assignments.patch before you push it ?

Comment 50

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Bob Clary [:bc:] from comment #49)

Yes - will do.

Comment 51

[Robert Wood [:rwood]]

Comment on attachment 9007097 [details] [diff] [review]
support apk install, logcat, etc

Review of attachment 9007097 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome thanks!

::: testing/mozharness/mozharness/mozilla/testing/android.py
@@ +99,5 @@
> +        install_ok = False
> +        if int(self.sdk_level) >= 23:
> +            cmd = ['install', '-r', '-g', self.installer_path]
> +        else:
> +            cmd = ['install', '-r', self.installer_path]

nit: maybe log a message here indicating that we are unable to set permissions on the android app at install time. Just in case down the road we have failures caused by permissions not set.

@@ +139,5 @@
> +        )
> +        if not os.path.exists(manifest_path):
> +            self.fatal("Could not retrieve manifest needed to retrieve "
> +                       "artifacts from %s" % manifest_path)
> +        self.tooltool_fetch(manifest_path,

nit: suggest add a comment here indicating that tooltool fetch is found in the TestingMixin (since there is no import/inheritance for it).

Comment 52

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: support apk install, logcat, etc - updated for review comments

Comment 53

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: p2-raptor-test-platform-assignments.patch

Updated to not change tier again.

Comment 54

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/90884f7b97d8
add test platform for android-hw-g5-7-0-arm7-api-16/opt - android-hw-arm7-raptor, r=gbrown,rwood
https://hg.mozilla.org/integration/mozilla-inbound/rev/50259b3caa6b
add test set for android-hw-arm7-raptor - raptor-speedometer-geckoview, r=gbrown,rwood.
https://hg.mozilla.org/integration/mozilla-inbound/rev/d4be0ae6ae4e
add raptor-speedometer-geckoview for android-hw-g5 test platforms, r=gbrown,rwood.
https://hg.mozilla.org/integration/mozilla-inbound/rev/56d714aedeb0
add raptor-speedometer-geckoview for android-hw-p2 test platforms, r=gbrown,rwood.
https://hg.mozilla.org/integration/mozilla-inbound/rev/638d02ffcc6f
add test platform for android-hw-p2-8-0-*/opt - android-hw-*-raptor, r=gbrown,rwood
https://hg.mozilla.org/integration/mozilla-inbound/rev/1d75f4dbb1e8
add test set for android-hw-aarch64-raptor - raptor-speedometer-geckoview, r=gbrown,rwood.
https://hg.mozilla.org/integration/mozilla-inbound/rev/251d13633524
Add geckoview installation and android diagnostics to raptor; r=bc,rwood

Comment 55

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/90884f7b97d8
https://hg.mozilla.org/mozilla-central/rev/50259b3caa6b
https://hg.mozilla.org/mozilla-central/rev/d4be0ae6ae4e
https://hg.mozilla.org/mozilla-central/rev/56d714aedeb0
https://hg.mozilla.org/mozilla-central/rev/638d02ffcc6f
https://hg.mozilla.org/mozilla-central/rev/1d75f4dbb1e8
https://hg.mozilla.org/mozilla-central/rev/251d13633524