1490451 - Add speedometer benchmark on raptor-browsertime for android google chrome

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P2)

Comment 1

[Geoff Brown [:gbrown]]

Let's make some decisions...

1. Do we want a new taskcluster task, or can we re-use "raptor-speedometer-chrome"?

https://hg.mozilla.org/integration/mozilla-inbound/file/tip/taskcluster/ci/test/raptor.yml#l56

raptor-speedometer-firefox:
    description: "Raptor Speedometer on Firefox"
    try-name: raptor-speedometer-firefox
...
raptor-speedometer-geckoview:
    description: "Raptor Speedometer on Geckoview"
    try-name: raptor-speedometer-geckoview
...
raptor-speedometer-chrome:
    description: "Raptor Speedometer on Chrome"
    try-name: raptor-speedometer-chrome
    treeherder-symbol: Rap-C(sp)
    run-on-projects: ['try', 'mozilla-central']
    max-run-time: 1500
    mozharness:
        extra-options:
            - --test=raptor-speedometer
            - --app=chrome

raptor-speedometer-chrome-android:
   description: "Raptor Speedometer on Chrome, on Android"
   try-name: raptor-speedometer-chrome-android
   treeherder-symbol: Rap-C(sp)
   run-on-projects: ['try', 'mozilla-central']
   max-run-time: 1500
   mozharness:
       extra-options:
           - --test=raptor-speedometer
           - --app=chrome
           - --binary=org.google.chrome

vs. something like

raptor-speedometer-chrome:
    description: "Raptor Speedometer on Chrome"
    try-name: raptor-speedometer-chrome
    treeherder-symbol: Rap-C(sp)
    run-on-projects: ['try', 'mozilla-central']
    max-run-time: 1500
    mozharness:
        extra-options:
            by-test-platform:
                android-hw.*:
                    - --test=raptor-speedometer
                    - --app=chrome
                    - --binary=org.google.chrome
                default:
                    - --test=raptor-speedometer
                    - --app=chrome

Comment 2

[Geoff Brown [:gbrown]]

In the raptor harness, we currently use the 'app' option to distinguish between android and desktop:

https://dxr.mozilla.org/mozilla-central/rev/fea371cafd2c73be689184ca6670e33c3fd1636c/testing/raptor/raptor/raptor.py#79

       if self.config['app'] == "geckoview":
            # create the android device handler; it gets initiated and sets up adb etc
            self.log.info("creating android device handler using mozdevice")
            self.device = ADBAndroid(verbose=True)
            self.device.clear_logcat()

What's the new condition, to include chrome-on-android?

1. Keep
        if self.config['app'] == "geckoview":
and use self.config['binary'] to distinguish geckoview/chrome.

2.         if self.config['app'] == "geckoview" or
              self.config['app'] == "chrome-android"   # "chrome" reserved for desktop
?

3. A new option?

Comment 3

[Robert Wood [:rwood]]

Attachment: Bug 1490451 - Add speedometer benchmark on raptor for android google chrome (wip)

Comment 4

[Robert Wood [:rwood]]

^ Preliminary patch. Invoke with:

./mach raptor-test --test raptor-speedometer --app=chrome-android --binary="com.android.chrome"

With my Google Pixel 2 (and the Chrome browser app already installed on it), Chrome is launched, but then raptor times out. I need to do some work in the raptor webextension for running on chrome android (hopefully it supports all the webext APIs that chrome desktop does).

Comment 5

[Robert Wood [:rwood]]

Note: We may be able to get the android chrome app from the same place Raptor is grabbing the desktop chromium from. i.e. At least getting the latest revision of android seems to work [1].

Figure out the rest of the URL and zip/apk file name and update that in mozharness at [2].

[1] http://commondatastorage.googleapis.com/chromium-browser-snapshots/Android/LAST_CHANGE

[2] https://searchfox.org/mozilla-central/rev/807a37c670c093b6e5201841a7c5315ba67ba8d5/testing/mozharness/mozharness/mozilla/testing/raptor.py#165

Comment 6

[Robert Wood [:rwood]]

The Google Chrome app on android doesn't officially support browser extensions :( Hacking around to confirm this / see if I can get it working anyway.

Comment 7

[Robert Wood [:rwood]]

Attachment: Bug 1490451 - Add speedometer benchmark on raptor for android google chrome (WIP)

Comment 8

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #7)
> Created attachment 9017290 [details] Bug 1490451 - Add speedometer benchmark on raptor for android google chrome (WIP)

Invoke with:

./mach raptor-test --test raptor-speedometer --app=chrome-android --binary="com.android.chrome"

Current state, Chrome app starts on the GP2 with "about:blank" loaded, then raptor hangs. Chrome app doesn't return error code on startup.

16:04:33     INFO -  adb launch_application: am start -W -n com.android.chrome/com.google.android.apps.chrome.Main --es --profile-directory Default --es --user-data-dir /sdcard/raptor-profile --es --load-extension /sdcard/raptor-profile/addons/raptor -d about:blank
16:04:34     INFO -  adb shell_output: adb -s FA79F1A05596 wait-for-device shell am start -W -n com.android.chrome/com.google.android.apps.chrome.Main --es --profile-directory Default --es --user-data-dir /sdcard/raptor-profile --es --load-extension /sdcard/raptor-profile/addons/raptor -d about:blank; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { dat=about:blank cmp=com.android.chrome/com.google.android.apps.chrome.Main (has extras) }
16:04:34     INFO -  Status: ok
16:04:34     INFO -  Activity: com.android.chrome/com.google.android.apps.chrome.Main
16:04:34     INFO -  ThisTime: 181
16:04:34     INFO -  TotalTime: 181
16:04:34     INFO -  WaitTime: 192
16:04:34     INFO -  Complete

When I debug the android Chrome app ('adb forward tcp:9222 localabstract:chrome_devtools_remote', then go to localhost:9222 in desktop Chrome) there are no errors in console - however there are no raptor webext messages either so looks like indeed the webext is not loading.

Comment 9

[Robert Wood [:rwood]]

Update: Yep the raptor webextension won't load on the google chrome android app, using the same cmd line opts that work on chrome desktop i.e.

11:31:21     INFO -  adb launch_application: am start -W -n com.android.chrome/com.google.android.apps.chrome.Main --es args '--load-extension=/sdcard/raptor-profile/addons/raptor' -d about:blank

Not really surprised since extensions on chrome android aren't supported. So we need to come up with an alternative.

One option that :jmaher mentioned on irc is maybe we can bypass the raptor webext when running on google chrome android and drive it explicitly via the benchmark url/url params. Raptor gets all test settings - test url, cycles, timeout, etc from within the webextension (via control server) so that would require some creativity to get around, but I will do some hacking to see what's involved going that route.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume we want to wait here until the speedometer tests can be run with browsertime. Having that test running under that tool, it will be much easier to get the job running for Chrome on Android.

Support for that benchmark and Firefox just got landed today via bug 1601009. I assume getting Chrome to run too is just a matter of the TaskCluster config?

Comment 11

[Florin Strugariu [:Bebe]]

If we can run browsertime tp6 tests on Chrome on Android then we can run speedometer too.
Just add it in taskcluster.

Currently I don't see any Chrome on Android for browser-time tests.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Browsertime uses chromedriver to interact with Chrome. Same applies to Android when chromedriver runs on the host machine (see http://chromedriver.chromium.org/getting-started/getting-started---android). With that we won't need a webextension anymore.

Dave, I assume we are better off morphing this bug to raptor-browsertime?

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This will be done via raptor-browsertime. It depends on bug 1598059.