Closed Bug 1615257 Opened 6 months ago Closed 4 months ago

Enable Chrome Browsertime pageload tests on Android

Categories

(Testing :: Raptor, task, P1)

Version 3
task

Tracking

(firefox76 fixed)

RESOLVED FIXED
mozilla76
Tracking Status
firefox76 --- fixed

People

(Reporter: sparky, Assigned: sparky)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

This bug is for adding chrome for android to the browsers we test with on Browsertime. This would involve building up a fetch task to get the Chrome for Android apk, and installing it on the device.

With that in place, we can start by adding 2-3 pageload tests running on the general_perf_testing cron task (it should pick them up already): https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#454

I've started working on this bug. Here's the patch I have in testing right now: https://hg.mozilla.org/try/rev/7ffef7da2aa3b336bf274091c21862b09cec17d4

Some questions that we need to figure out are:

(1) Can we do manual chrome upgrades rather than automated ones? They would be done at the same time as the other chrome upgrades that we do so it's not adding anything to the process we already have.

(2) Name of the android chrome browser is chrome-m because the perfherder schema limits the number of characters to 10. Should we increase this and make a better name?

(3) What tests should we start with in CI? I am thinking we should land this with amazon and youtube tests running in the general perf-testing cron task. We can increase the tests afterwards.

Assignee: nobody → gmierz2
Status: NEW → ASSIGNED
Flags: needinfo?(dave.hunt)
Priority: P2 → P1

Also, I think that bug 1598059 is an old bug for this work. I think we should set is as a duplicate of this bug.

See Also: → 1598059

So with the chrome download we're running into something interesting where the download page is a PHP file and it prevents downloads if there is no user-agent. Using curl seemed to get around this issue locally so I'm testing this in CI now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4487186fc465bea1bd0e7eb14a888312bc9955e5

It might be best if we get the Chrome APKs from this website, and then upload them to tooltool and download/install from there to prevent issues in the future.

(In reply to Greg Mierzwinski [:sparky] from comment #2)

Also, I think that bug 1598059 is an old bug for this work. I think we should set is as a duplicate of this bug.

It's a meta bug, so lets block it for now. Maybe some follow-up work is necessary after the patch on this bug landed.

Blocks: 1598059
See Also: 1598059

(In reply to Greg Mierzwinski [:sparky] from comment #1)

I've started working on this bug. Here's the patch I have in testing right now: https://hg.mozilla.org/try/rev/7ffef7da2aa3b336bf274091c21862b09cec17d4

Some questions that we need to figure out are:

(1) Can we do manual chrome upgrades rather than automated ones? They would be done at the same time as the other chrome upgrades that we do so it's not adding anything to the process we already have.

We can do this in the short term, but worth looping in :aerickson as would need to be added to any device provisioning procedures that we have. Ultimately I'd like us to automate this.

(2) Name of the android chrome browser is chrome-m because the perfherder schema limits the number of characters to 10. Should we increase this and make a better name?

Can we just use "chrome"? We can use the platform to distinguish that this is the mobile browser.

(3) What tests should we start with in CI? I am thinking we should land this with amazon and youtube tests running in the general perf-testing cron task. We can increase the tests afterwards.

This sounds like a reasonable place to start.

Flags: needinfo?(dave.hunt)

I don't think we'll be able to automate this unless Google releases something to help with that. Right now, we have to go through third-parties to get the APKs to install. None of these sources provide a simple way to find the latest version. That said, we could probably do some webpage scraping to get that but this won't be simple. I'll keep :aerickson in mind as I move forward with this though.

Unfortunately, we can't use "chrome" because it's reserved for the desktop version. Making the distinction with the platform won't help with this issue either because our test manifests require an app name to get the test settings. If we use chrome, then we'll have android tests enabled for desktop chrome. There's a lot of other things that are desktop/android specific based on the app name that will need to changed and it will get very hacky very quickly. Because of these issues, we would have to make some large changes to get this minor issue solved and I don't think it's worth the effort.

(In reply to Greg Mierzwinski [:sparky] from comment #6)

I don't think we'll be able to automate this unless Google releases something to help with that. Right now, we have to go through third-parties to get the APKs to install. None of these sources provide a simple way to find the latest version. That said, we could probably do some webpage scraping to get that but this won't be simple. I'll keep :aerickson in mind as I move forward with this though.

Unfortunately, we can't use "chrome" because it's reserved for the desktop version. Making the distinction with the platform won't help with this issue either because our test manifests require an app name to get the test settings. If we use chrome, then we'll have android tests enabled for desktop chrome. There's a lot of other things that are desktop/android specific based on the app name that will need to changed and it will get very hacky very quickly. Because of these issues, we would have to make some large changes to get this minor issue solved and I don't think it's worth the effort.

Ugh.. can we at least present this as "chrome" in Perfherder and our dashboards?

Yes, good idea! That will be easy to do, and I'm guessing the dashboards don't use the app names for anything other than displaying them?

Turns out that we've already got the latest version of Chrome on the devices in CI! Plus, they auto-update so that's a bonus. :)

(In reply to Greg Mierzwinski [:sparky] from comment #9)

Turns out that we've already got the latest version of Chrome on the devices in CI! Plus, they auto-update so that's a bonus. :)

Are we able to query the version installed and include it in the perfherder.json file? This would be useful for quickly identifying when a change is related to an application update. It also provides a quick method of confirming we're using the correct latest version, or for looking back at historical results.

Yup, I implemented that yesterday: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=134e730fa490263a90d5c988a9fa23ab18955097

We need to have this in place to be able to find the correct chromedriver that should be used.

This patch adds the capability to run Google Chrome for Android tests through Raptor-Browsertime.

Chrome must be available on the device being tested, and there are no installation steps being added in this patch since CI already has the latest Chrome version available to test with. That said, this patch adds the ability to find the version of Chrome that is being tested and stores this in the Perfherder artifact. Getting this version is also necessary to be able to run Chrome with the correct chromedriver.

Two tests are initially be enabled for Chrome in this patch: Amazon, and YouTube. They will only run through a cron task three days a week. The other changes done in this patch are required for Chrome to work with Raptor-Browsertime.

Blocks: 1625470
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/07d40de55dbb
Enable google chrome android browsertime tests. r=perftest-reviewers,AlexandruIonescu
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla76
You need to log in before you can comment on or make changes to this bug.