Closed Bug 1511356 Opened Last year Closed 11 months ago

Run Speedometer tests against Fennec (ARMv7 and ARM64)

Categories

(Testing :: Raptor, defect)

Version 3
Unspecified
Android
defect
Not set

Tracking

(firefox66 fixed)

RESOLVED FIXED
mozilla66
Tracking Status
firefox66 --- fixed

People

(Reporter: davehunt, Assigned: rwood)

References

Details

(Whiteboard: [geckoview:fenix:p1])

Attachments

(1 file)

We'd like to add running Speedometer on Fennec so we can track performance improvements in the Android Components reference browser.
:rwood could you take a look into what would be involved here? These most likley wouldn't need to run frequently or into the long-term, so maybe there are some shortcuts we could take to get some initial results?
Flags: needinfo?(rwood)
We've never ran Raptor on Fennec before, it all depends on Fennec's web extension support and if the Raptor web extension is fully supported. The good news is that we don't need a CA certificate or the proxy settings to run Speedometer, so that makes it faster than adding say pageload tp6 to Fennec. Sorry I can't really estimate how long until I actually try it out and see how well the webext works.

Not really any shortcuts - except if numbers are needed ASAP then Speedometer could be run manually on Fennec at least to get some initial numbers.
Flags: needinfo?(rwood)
:sphilp are you okay with the Speedometer score to be run manually on our reference hardware, at least for the short term?
Flags: needinfo?(sphilp)
Yes manual is fine for now. Getting this running some time in the coming weeks would be great but not super urgent.
Flags: needinfo?(sphilp)
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Blocks: 1473078
(In reply to Robert Wood [:rwood] from comment #6)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d4b31164f63a4e8e78d14bf264c5c7881fbc1ad

Looks like the android nightly build/test jobs aren't setup (?) :dustin, is there anyone who could please have a look at this or help me get this job running on a fennec nightly?
Flags: needinfo?(dustin)
Hm, this is definitely out of my wheelhouse, enough that I'm not even understanding what the problem is.  That said, for fennec nightlies, I'm guessing Johan may be at least a step closer to the person who can help..
Flags: needinfo?(dustin) → needinfo?(jlorenzo)
Hey Rob!

Good news first, I tested your changes against the nightly graph, and these tasks will be part of the nightly desktop graph: 
* test-android-hw-p2-8-0-arm7-api-16-nightly/opt-raptor-speedometer-fennec
* test-android-hw-p2-8-0-android-aarch64-nightly/opt-raptor-speedometer-fennec

(If you want to reproduce this locally:)
> 1. Download a parameters.yml file from a nightly android decision task[1]
> 2. ./mach taskgraph optimized --json --parameters parameters.yml > decision-output.json
> 3. Search through the generated json file to find the raptor tasks



Then, regarding the error, it's caused by the fact that Nightly artifacts aren't exposed the same way as on a regular build. In extenso, Nightly exposes public/build/en-US/target.common.tests.tar.gz[2], but regular build has public/build/target.common.tests.tar.gz[3]. To me, this has never happened before because nightly doesn't run any tests (with the exception of linting)[4]. 
Root cause: public/build/target.test_packages.json is busted. It reports artifacts as if they were from a regular build.

Per [5], :chmanchester may be able to help. What do you think, Chris? 

[1] For instance: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=nightly&selectedJob=216161036&revision=3386ff76878d83496bb822d09115c77472808b53
[2] E.g.: https://tools.taskcluster.net/groups/TYhw2IDoTs2zZn7DkzLMZw/tasks/HKsEq1MDQP2qnFpUJ6uCjw/runs/0/artifacts
[3] Like: https://tools.taskcluster.net/groups/H6S_nSPwTbCHDVxj9uCJJw/tasks/BnMXl7_8QEGDozxQ6vnexg/runs/0/artifacts
[4] For instance: https://tools.taskcluster.net/groups/TYhw2IDoTs2zZn7DkzLMZw
[5] https://searchfox.org/mozilla-central/rev/dc3585e58b427c3a8a7c01712fe54ebe118a3ce2/build/gen_test_packages_manifest.py
Flags: needinfo?(jlorenzo) → needinfo?(cmanchester)
target.test_packages.json doesn't contain full paths, I don't think it's to blame.

I suspect there's code in mozharness that handles locating artifacts, but my knowledge is way out of date here.
Flags: needinfo?(cmanchester)
Okay. Should target.test_packages.json contain relative paths? So far, I've seen it containing file names only.
Flags: needinfo?(cmanchester)
I believe it's working as intended containing file names only. I expect there's some code in mozharness that locates the path to the artifacts (perhaps via the taskcluster index).
Flags: needinfo?(cmanchester)
We're also going to need to run page load tests on Fennec so we can compare with Fenix. What's the next steps to getting this resolved?
Flags: needinfo?(jlorenzo)
(In reply to Chris Manchester (:chmanchester) from comment #12)
> I believe it's working as intended containing file names only. I expect
> there's some code in mozharness that locates the path to the artifacts
> (perhaps via the taskcluster index).

I'm not too familiar with Mozharness. From what I see, the URL is requested at [1], which is taken directly from test-package-url[2], which itself is provided by a command-line argument[3], which in this test is[4]:

> 20:45:50     INFO - Run as /builds/worker/workspace/mozharness/scripts/raptor_script.py --config-file /builds/worker/workspace/mozharness/configs/raptor/android_hw_config.py --installer-url=https://queue.taskcluster.net/v1/task/LvQr0nT0Tz21tRfISnJLbA/artifacts/public/build/target.apk --test-packages-url=https://queue.taskcluster.net/v1/task/LvQr0nT0Tz21tRfISnJLbA/artifacts/public/build/target.test_packages.json --test=raptor-fennec --app=fennec --binary=org.mozilla.fennec_aurora --download-symbols=ondemand

The file name is directly extracted from target.test_packages.json[5]. If we wanted to make mozharness smarter, we would need a way to detect somewhere in _download_test_packages() if we're in the regular case, or in the nightly one. Geoff, I see you've done most of the changes in this function. Would changing mozharness to detect this edge case make sense to you? 

[1] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#372
[2] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#129
[3] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#80
[4] https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215819118&repo=try&lineNumber=395
[5] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#352
[6] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/testing/mozharness/mozharness/mozilla/testing/testbase.py#141
Flags: needinfo?(jlorenzo) → needinfo?(gbrown)
I don't oppose updating _download_test_packages(), but I don't know how it would determine if it was handling a nightly case or not.

Looking at this from a different angle, the issue is that there's a mismatch between the installer-url and the test packages url(s). I don't know much about nightly builds, but I notice that they seem to have 2 target.apk artifacts - one at .../public/build/target.apk and one at .../public/build/en-US/target.apk. If we used an installer-url of .../public/build/en-US/target.apk (with some sort of taskcluster modification), then the existing mozharness logic would find the test packages. Is that an option?
Flags: needinfo?(gbrown)
Great idea! Rob, is using artifacts under public/build/en-US good for your case? If so, you can update [1] to output a different path if it's a nightly. What do you think? 

[1] https://searchfox.org/mozilla-central/rev/1a4a7745fd4a14402a0de0e9abc040724d4fcf8f/taskcluster/taskgraph/transforms/job/mozharness_test.py#54
Flags: needinfo?(rwood)
I stumbled on bug 1486128, which implies that maybe there should be only one target.apk - interesting.
See Also: → 1486128
(In reply to Johan Lorenzo [:jlorenzo] from comment #16)
> Great idea! Rob, is using artifacts under public/build/en-US good for your
> case? If so, you can update [1] to output a different path if it's a
> nightly. What do you think? 
> 
> [1]
> https://searchfox.org/mozilla-central/rev/
> 1a4a7745fd4a14402a0de0e9abc040724d4fcf8f/taskcluster/taskgraph/transforms/
> job/mozharness_test.py#54

Thanks guys, I'll give that a go.
Flags: needinfo?(rwood)
Adding [geckoview:fenix:p1] whiteboard tag because measuring Fennec performance (for both ARMv7 and ARM64) is needed for Fenix 1.0's release criteria.
OS: Unspecified → Android
Summary: Run Speedometer tests against Fennec → Run Speedometer tests against Fennec (ARMv7 and ARM64)
Whiteboard: [geckoview:fenix:p1]
(In reply to Robert Wood [:rwood] from comment #20)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b2c87a7f78e17e2e5e25f86c1261d508ee3502

Awesome, green on try.

:wcosta, I see your name in the history for /taskcluster/taskgraph/transforms/job/mozharness_test.py. Would you mind reviewing my changes to that file in this patch? (attachment 9029868 [details]) i.e:

https://phabricator.services.mozilla.com/D13946#change-4dJloBrBxPoo
Flags: needinfo?(wcosta)
(In reply to Robert Wood [:rwood] from comment #22)
> (In reply to Robert Wood [:rwood] from comment #20)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b2c87a7f78e17e2e5e25f86c1261d508ee3502
> 
> Awesome, green on try.
> 
> :wcosta, I see your name in the history for
> /taskcluster/taskgraph/transforms/job/mozharness_test.py. Would you mind
> reviewing my changes to that file in this patch? (attachment 9029868 [details]
> [details]) i.e:
> 
> https://phabricator.services.mozilla.com/D13946#change-4dJloBrBxPoo

It has been a long time since I touched mozharness, :jlund may be a better person to review or indicate someone to do it.
Flags: needinfo?(wcosta)
Thanks Wander. :jlund, would you mind a quick review (comment 22) please (or suggest whom), thanks!
Flags: needinfo?(jlund)
(In reply to Robert Wood [:rwood] from comment #24)
> Thanks Wander. :jlund, would you mind a quick review (comment 22) please (or
> suggest whom), thanks!


(we are chatting in phabricator)
Flags: needinfo?(jlund)
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8ad51093f3f0
Run speedometer on fennec via Raptor; r=jmaher,jlund
https://hg.mozilla.org/mozilla-central/rev/8ad51093f3f0
Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
You should probably do the treeherder platform name map stuff as well so you don't get the android-hw-p2-8-0-arm7-api-16-nightly opt and android-hw-p2-8-0-android-aarch64-nightly opt names. There should be some way to distinguish Fennec from other apps as well. Perhaps by group or job symbol? See bug 1469755 and bug 1472206.
Depends on: 1517738
(In reply to Bob Clary [:bc:] from comment #30)
> You should probably do the treeherder platform name map stuff as well so you
> don't get the android-hw-p2-8-0-arm7-api-16-nightly opt and
> android-hw-p2-8-0-android-aarch64-nightly opt names. There should be some
> way to distinguish Fennec from other apps as well. Perhaps by group or job
> symbol? See bug 1469755 and bug 1472206.

Some Android emulator unit tests run on Fennec; others in the geckoview test app. They can be distinguished by the group's e10s-ness: Fennec == non-e10s; geckoview == e10s. Obviously that's not explicit, and that rule could break down in future. If we adopt more explicit labels for android-hw, I'd like to see the same for android-em (keep all android consistent).
You need to log in before you can comment on or make changes to this bug.