1613455 - Support "push-to-try" for Fenix performance tests

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P3)

Description

[Andrew Creskey [:acreskey]]

Currently, browsertime in automation will use a fixed binary for geckoview-consuming apps like Fenix (performanceTest nightly, I believe).

So this doesn't provide a means to evaluate the performance impact of changes to either Gecko or Fenix source.

The bug is to provide a method by which the user can provide the .apk used in the test (perhaps simply a url to the apk).

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1613455 - [WIP] Allow custom APKs to be installed by raptor.

Comment 2

[Greg Mierzwinski [:sparky]]

:acreskey, the patch I posted works locally and lets you install/re-install custom APKs of the Firefox Android browsers (using --installer-path and --binary). But it's quite likely that you'll hit some unexpected errors while using this patch.

I thought this would have been a small fix but I think there's a lot more to do here (mainly because the mozharness+mach intricacies). See bug 1616960.

Comment 3

[Andrew Creskey [:acreskey]]

Thank you for moving this along Greg. If you don't mind, I'm going to way for this in automation because local runs though ./mach are not something I do frequently.

Comment 4

[Greg Mierzwinski [:sparky]]

I just want to mention that we've decided to leave this as a P3 since there's nothing crucial that depends on this at the moment. If that's not the case, please say so, otherwise, we will hack on this later on.

Comment 5

[Andrew Creskey [:acreskey]]

I've renamed this to make it more clear.
What we would be looking for is the ability to "push to try" changes in gecko, but testing them in Fenix.
This would be useful for performance experiments (startup, pageload, some day responsiveness).

Comment 6

[Andrew Creskey [:acreskey]]

One complication is that a given mozilla-central is not always API compatible with the latest Android-Components and Fenix.

It may be necessary to "push-to-try" but also select a compatible Fenix revision.

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1613455 - Allow custom APK uploads for Geckoview/Fenix perftests. r?#perftest

This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.

The ./mach try perf command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the ./mach try perf command to select the appropriate tests.

Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/faa1c03828fe
Allow custom APK uploads for Geckoview/Fenix perftests. r=perftest-reviewers,afinder

Comment 9

[Iulian Moraru]

Backed out for causing btime failures.

• Push with failures

• Push where this started failing

• Failure log

• Backout link

[task 2023-03-22T16:51:53.713Z] 16:51:47     INFO - Return code: 0
[task 2023-03-22T16:51:53.713Z] Using adb 1.0.41
[task 2023-03-22T16:51:53.713Z] /system/bin/ls -1A supported
[task 2023-03-22T16:51:53.713Z] Native cp support: True
[task 2023-03-22T16:51:53.713Z] Native chmod -R support: True
[task 2023-03-22T16:51:53.713Z] Native chown -R support: True
[task 2023-03-22T16:51:53.713Z] Native normal pidof support: True
[task 2023-03-22T16:51:53.713Z] adbd not restarted as root
[task 2023-03-22T16:51:53.713Z] su -c setenforce 0 exitcode 0, stdout: None
[task 2023-03-22T16:51:53.713Z] su -c supported
[task 2023-03-22T16:51:53.713Z] Setting SELinux Permissive
[task 2023-03-22T16:51:59.029Z] 16:51:53     INFO - Running post-action listener: _resource_record_post_action
[task 2023-03-22T16:51:59.029Z] 16:51:53     INFO - [mozharness: 2023-03-22 16:51:53.641665Z] Finished install step (failed)
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -   File "/builds/task_167950380049590/workspace/mozharness/mozharness/base/script.py", line 2385, in run
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -     self.run_action(action)
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -   File "/builds/task_167950380049590/workspace/mozharness/mozharness/base/script.py", line 2319, in run_action
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -   File "/builds/task_167950380049590/workspace/mozharness/mozharness/base/script.py", line 2271, in _possibly_run_method
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -     return getattr(self, method_name)()
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -   File "/builds/task_167950380049590/workspace/mozharness/mozharness/mozilla/testing/raptor.py", line 1246, in install
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL -     installer_path = self.installerpath
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL - AttributeError: 'Raptor' object has no attribute 'installerpath'
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL - Running post_fatal callback...
[task 2023-03-22T16:51:59.029Z] 16:51:53    FATAL - Exiting -1

Comment 10

[Greg Mierzwinski [:sparky]]

Thanks for backing out, I've updated the patch and re-requested review.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d44809bdb073
Allow custom APK uploads for Geckoview/Fenix perftests. r=perftest-reviewers,afinder

Comment 12

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/d44809bdb073