1511356 - Run Speedometer tests against Fennec (ARMv7 and ARM64)

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We'd like to add running Speedometer on Fennec so we can track performance improvements in the Android Components reference browser.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:rwood could you take a look into what would be involved here? These most likley wouldn't need to run frequently or into the long-term, so maybe there are some shortcuts we could take to get some initial results?

Comment 2

[Robert Wood [:rwood]]

We've never ran Raptor on Fennec before, it all depends on Fennec's web extension support and if the Raptor web extension is fully supported. The good news is that we don't need a CA certificate or the proxy settings to run Speedometer, so that makes it faster than adding say pageload tp6 to Fennec. Sorry I can't really estimate how long until I actually try it out and see how well the webext works.

Not really any shortcuts - except if numbers are needed ASAP then Speedometer could be run manually on Fennec at least to get some initial numbers.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:sphilp are you okay with the Speedometer score to be run manually on our reference hardware, at least for the short term?

Comment 4

[Stuart Philp :sphilp]

Yes manual is fine for now. Getting this running some time in the coming weeks would be great but not super urgent.

Comment 5

[Robert Wood [:rwood]]

Attachment: Bug 1511356 - Run speedometer on fennec via Raptor; r?jmaher,davehunt

Comment 6

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d4b31164f63a4e8e78d14bf264c5c7881fbc1ad

Comment 7

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #6)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d4b31164f63a4e8e78d14bf264c5c7881fbc1ad

Looks like the android nightly build/test jobs aren't setup (?) :dustin, is there anyone who could please have a look at this or help me get this job running on a fennec nightly?

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, this is definitely out of my wheelhouse, enough that I'm not even understanding what the problem is.  That said, for fennec nightlies, I'm guessing Johan may be at least a step closer to the person who can help..

Comment 9

[Johan Lorenzo [:jlorenzo]]

Hey Rob!

Good news first, I tested your changes against the nightly graph, and these tasks will be part of the nightly desktop graph: 
* test-android-hw-p2-8-0-arm7-api-16-nightly/opt-raptor-speedometer-fennec
* test-android-hw-p2-8-0-android-aarch64-nightly/opt-raptor-speedometer-fennec

(If you want to reproduce this locally:)
> 1. Download a parameters.yml file from a nightly android decision task[1]
> 2. ./mach taskgraph optimized --json --parameters parameters.yml > decision-output.json
> 3. Search through the generated json file to find the raptor tasks



Then, regarding the error, it's caused by the fact that Nightly artifacts aren't exposed the same way as on a regular build. In extenso, Nightly exposes public/build/en-US/target.common.tests.tar.gz[2], but regular build has public/build/target.common.tests.tar.gz[3]. To me, this has never happened before because nightly doesn't run any tests (with the exception of linting)[4]. 
Root cause: public/build/target.test_packages.json is busted. It reports artifacts as if they were from a regular build.

Per [5], :chmanchester may be able to help. What do you think, Chris? 

[1] For instance: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=nightly&selectedJob=216161036&revision=3386ff76878d83496bb822d09115c77472808b53
[2] E.g.: https://tools.taskcluster.net/groups/TYhw2IDoTs2zZn7DkzLMZw/tasks/HKsEq1MDQP2qnFpUJ6uCjw/runs/0/artifacts
[3] Like: https://tools.taskcluster.net/groups/H6S_nSPwTbCHDVxj9uCJJw/tasks/BnMXl7_8QEGDozxQ6vnexg/runs/0/artifacts
[4] For instance: https://tools.taskcluster.net/groups/TYhw2IDoTs2zZn7DkzLMZw
[5] https://searchfox.org/mozilla-central/rev/dc3585e58b427c3a8a7c01712fe54ebe118a3ce2/build/gen_test_packages_manifest.py

Comment 10

[Chris Manchester (limited bugmail, email directly)]

target.test_packages.json doesn't contain full paths, I don't think it's to blame.

I suspect there's code in mozharness that handles locating artifacts, but my knowledge is way out of date here.

Comment 11

[Johan Lorenzo [:jlorenzo]]

Okay. Should target.test_packages.json contain relative paths? So far, I've seen it containing file names only.

Comment 12

[Chris Manchester (limited bugmail, email directly)]

I believe it's working as intended containing file names only. I expect there's some code in mozharness that locates the path to the artifacts (perhaps via the taskcluster index).

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We're also going to need to run page load tests on Fennec so we can compare with Fenix. What's the next steps to getting this resolved?

Comment 14

[Johan Lorenzo [:jlorenzo]]

(In reply to Chris Manchester (:chmanchester) from comment #12)
> I believe it's working as intended containing file names only. I expect
> there's some code in mozharness that locates the path to the artifacts
> (perhaps via the taskcluster index).

I'm not too familiar with Mozharness. From what I see, the URL is requested at [1], which is taken directly from test-package-url[2], which itself is provided by a command-line argument[3], which in this test is[4]:

> 20:45:50     INFO - Run as /builds/worker/workspace/mozharness/scripts/raptor_script.py --config-file /builds/worker/workspace/mozharness/configs/raptor/android_hw_config.py --installer-url=https://queue.taskcluster.net/v1/task/LvQr0nT0Tz21tRfISnJLbA/artifacts/public/build/target.apk --test-packages-url=https://queue.taskcluster.net/v1/task/LvQr0nT0Tz21tRfISnJLbA/artifacts/public/build/target.test_packages.json --test=raptor-fennec --app=fennec --binary=org.mozilla.fennec_aurora --download-symbols=ondemand

The file name is directly extracted from target.test_packages.json[5]. If we wanted to make mozharness smarter, we would need a way to detect somewhere in _download_test_packages() if we're in the regular case, or in the nightly one. Geoff, I see you've done most of the changes in this function. Would changing mozharness to detect this edge case make sense to you? 

[1] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#372
[2] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#129
[3] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#80
[4] https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215819118&repo=try&lineNumber=395
[5] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#352
[6] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#141

Comment 15

[Geoff Brown [:gbrown]]

I don't oppose updating _download_test_packages(), but I don't know how it would determine if it was handling a nightly case or not.

Looking at this from a different angle, the issue is that there's a mismatch between the installer-url and the test packages url(s). I don't know much about nightly builds, but I notice that they seem to have 2 target.apk artifacts - one at .../public/build/target.apk and one at .../public/build/en-US/target.apk. If we used an installer-url of .../public/build/en-US/target.apk (with some sort of taskcluster modification), then the existing mozharness logic would find the test packages. Is that an option?

Comment 16

[Johan Lorenzo [:jlorenzo]]

Great idea! Rob, is using artifacts under public/build/en-US good for your case? If so, you can update [1] to output a different path if it's a nightly. What do you think? 

[1] https://searchfox.org/mozilla-central/rev/1a4a7745fd4a14402a0de0e9abc040724d4fcf8f/taskcluster/taskgraph/transforms/job/mozharness_test.py#54

Comment 17

[Geoff Brown [:gbrown]]

I stumbled on bug 1486128, which implies that maybe there should be only one target.apk - interesting.

Comment 18

[Robert Wood [:rwood]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #16)
> Great idea! Rob, is using artifacts under public/build/en-US good for your
> case? If so, you can update [1] to output a different path if it's a
> nightly. What do you think? 
> 
> [1]
> https://searchfox.org/mozilla-central/rev/
> 1a4a7745fd4a14402a0de0e9abc040724d4fcf8f/taskcluster/taskgraph/transforms/
> job/mozharness_test.py#54

Thanks guys, I'll give that a go.

Comment 19

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=037a63956eab9dcad2ef00cb33bd02f1edc73131

Comment 20

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b2c87a7f78e17e2e5e25f86c1261d508ee3502

Comment 21

[Chris Peterson [:cpeterson]]

Adding [geckoview:fenix:p1] whiteboard tag because measuring Fennec performance (for both ARMv7 and ARM64) is needed for Fenix 1.0's release criteria.

Comment 22

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #20)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b2c87a7f78e17e2e5e25f86c1261d508ee3502

Awesome, green on try.

:wcosta, I see your name in the history for /taskcluster/taskgraph/transforms/job/mozharness_test.py. Would you mind reviewing my changes to that file in this patch? (attachment 9029868 [details]) i.e:

https://phabricator.services.mozilla.com/D13946#change-4dJloBrBxPoo

Comment 23

[Wander Lairson Costa]

(In reply to Robert Wood [:rwood] from comment #22)
> (In reply to Robert Wood [:rwood] from comment #20)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b2c87a7f78e17e2e5e25f86c1261d508ee3502
> 
> Awesome, green on try.
> 
> :wcosta, I see your name in the history for
> /taskcluster/taskgraph/transforms/job/mozharness_test.py. Would you mind
> reviewing my changes to that file in this patch? (attachment 9029868 [details]
> [details]) i.e:
> 
> https://phabricator.services.mozilla.com/D13946#change-4dJloBrBxPoo

It has been a long time since I touched mozharness, :jlund may be a better person to review or indicate someone to do it.

Comment 24

[Robert Wood [:rwood]]

Thanks Wander. :jlund, would you mind a quick review (comment 22) please (or suggest whom), thanks!

Comment 25

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5264742baac11c784d02777a1e25f93db4546d8

Comment 26

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d96de8c646229b324d6dcfdb17ab3105fcef6720

Comment 27

[Jordan Lund (:jlund)]

(In reply to Robert Wood [:rwood] from comment #24)
> Thanks Wander. :jlund, would you mind a quick review (comment 22) please (or
> suggest whom), thanks!


(we are chatting in phabricator)

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8ad51093f3f0
Run speedometer on fennec via Raptor; r=jmaher,jlund

Comment 29

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/8ad51093f3f0

Comment 30

[Bob Clary [:bc] (inactive)]

You should probably do the treeherder platform name map stuff as well so you don't get the android-hw-p2-8-0-arm7-api-16-nightly opt and android-hw-p2-8-0-android-aarch64-nightly opt names. There should be some way to distinguish Fennec from other apps as well. Perhaps by group or job symbol? See bug 1469755 and bug 1472206.

Comment 31

[Geoff Brown [:gbrown]]

(In reply to Bob Clary [:bc:] from comment #30)
> You should probably do the treeherder platform name map stuff as well so you
> don't get the android-hw-p2-8-0-arm7-api-16-nightly opt and
> android-hw-p2-8-0-android-aarch64-nightly opt names. There should be some
> way to distinguish Fennec from other apps as well. Perhaps by group or job
> symbol? See bug 1469755 and bug 1472206.

Some Android emulator unit tests run on Fennec; others in the geckoview test app. They can be distinguished by the group's e10s-ness: Fennec == non-e10s; geckoview == e10s. Obviously that's not explicit, and that rule could break down in future. If we adopt more explicit labels for android-hw, I'd like to see the same for android-em (keep all android consistent).