1876822 - Make sure Chrome release is up to date before running it

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P2)

Description

[Jeff Muizelaar [:jrmuizel]]

This graph shows Chrome getting faster and then getting slower. https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=1&series=mozilla-central,4781260,1,13&series=mozilla-central,4592661,1,13&series=mozilla-central,4591346,1,13&series=autoland,4722974,1,13&timerange=1209600&zoom=1706097495603,1706301408277,9.37742388488218,13.707835695474756

This is caused by us running Chrome 120, Chrome 121 and then Chrome 120 again.

It would be good if we tried to ensure that Chrome was up to date before running it.

It seems like that might be possible with something like:
https://groups.google.com/a/chromium.org/g/chromium-dev/c/rnSua-ApnOg/m/gROMCpTDBwAJ

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

:markco (not sure if you're the best to ping for this)
would you know/have thoughts on this strategy of calling X:\Program Files (x86)\Google\Update\GoogleUpdate.exe /ua /installsource scheduler (as mentioned here)to keep the Windows Chrome binaries up to date?

My gut feeling is it may not be wise to do this within the raptor-browsertime scripts, but I could be wrong.
Is there a better/safer alternative perhaps on the relops side? (maybe a script that runs before the windows browsertime tests run?)

Comment 2

[Mark Cornmesser [:markco]]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #1)

:markco (not sure if you're the best to ping for this)
would you know/have thoughts on this strategy of calling X:\Program Files (x86)\Google\Update\GoogleUpdate.exe /ua /installsource scheduler (as mentioned here)to keep the Windows Chrome binaries up to date?

My gut feeling is it may not be wise to do this within the raptor-browsertime scripts, but I could be wrong.
Is there a better/safer alternative perhaps on the relops side? (maybe a script that runs before the windows browsertime tests run?)

Sorry that I missed this last week. It's kind of tough to say here. The Chrome updater runs automatically so that we don't have to rely on code changes to keep Chrome updated. Because of that, I think we are ending up in 1 of 3 states. 1. Chrome is up-to-date. 2. Chrome is actively updating. Chrome has been updated, and the browser needs to be restarted.

Because the update service is being run automatically, the GoogleUpdate.exe is running when the browser is launched. I tested running GoogleUpdate.exe /ua /installsource scheduler. It seems to have not made a difference. The browser still started and then needed a restart to be updated.

The worker is unaware of what task it is going to run next, so we are unable to run anything before a specific task. The path of least resistance might be to launch, wait a short time, and close/relaunch the browser.

I will take some time this week and see if i can figure out another solution.

Comment 3

[Mark Cornmesser [:markco]]

I haven't been able to find any other solutions,

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

That's unfortunate, but thanks for checking :markco!

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

Also took a look at the browsertime repo doesn't look like there's anything that might help.

A potential hacky solution we can try out:

when running on Windows, use the JSON endpoint here https://googlechromelabs.github.io/chrome-for-testing/#stable
add some sort of check here to see of the chrome version matches the latest stable from the JSON endpoint. And if not, start, kill, and restart chrome and verify if the version is updated

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

Another option might be Chocolatey? e.g. If it is installed, would running choco install google chrome as a subprocess in our raptor scripts be safe? And does it behave in a way where it would always check the version and update, and do nothing otherwise? I see browsertime does this in the Windows workflow

Though, I am not familiar with Windows package management

:markco Do you know if choco is installed on the windows machine/ if something like this is safe to do?

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

Forgot the ni? in the previous comment ^

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #6)

Another option might be Chocolatey? e.g. If it is installed, would running choco install google chrome as a subprocess in our raptor scripts be safe? And does it behave in a way where it would always check the version and update, and do nothing otherwise? I see browsertime does this in the Windows workflow

Though, I am not familiar with Windows package management

:markco Do you know if choco is installed on the windows machine/ if something like this is safe to do?

Or scoop?

Comment 9

[Mark Cornmesser [:markco]]

Taking a look, Chocolatey is the better option. It looks like that package is maintained and up to date. The Scoop package looks outdated. This is unfortunate since Scoop works without admin rights.

We could test removing Chrome from the config and adding Chocolatey. However, it will require elevated privileges to install packages. Which I believe these workers have the Taskcluster scope to do.

Currently, we are in the process of setting up new hardware. We can take a look at having a test pool up with Chocolatey after the rollout.

https://mozilla-hub.atlassian.net/browse/RELOPS-872.

Comment 10

[Kash Shampur [:kshampur] ⌚EST]

great, thanks for the update!

Comment 11

[:jmoss]

After doing some testing, there is not a reliable way to launch google chrome using the methods shown in the google group forum. We have 2 options:

1. Leave as is.
2. Update the worker pool monthly with the latest version of google chrome that is available at that time.

In looking at the documentation for browsertime I noticed that there is a possibility of using ChromeDriver.exe instead of fully installing Chrome (more here). We could potentially look into having that be automatically installed, but this may not be a viable option.

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

Thanks for the update! option 1 is probably best for now. (2) would be good if the chrome updates came regularly on a monthly schedule but there seems to always be a delay. e.g. this roadmap is not always on time in my experience.

In looking at the documentation for browsertime I noticed that there is a possibility of using ChromeDriver.exe instead of fully installing Chrome (more here).

Could you clarify what you mean about using chromedriver.exe instead of installing Chrome?
Looking at the browserstack link, one eventual goal related to this bug was to automate grabbing the latest (stable) chromedrivers from chrome-for-testing (CfT) dashboard but that would require our Windows Chrome to be reliably be up to date (like it is on linux/mac for the most part)

worth also mentioning, we unfortunately can't use the Chrome binaries from the CfT dashboard ^ as they are not built with PGO so we can't rely on those performance numbers.

Comment 13

[:jmoss]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #12)

Thanks for the update! option 1 is probably best for now. (2) would be good if the chrome updates came regularly on a monthly schedule but there seems to always be a delay. e.g. this roadmap is not always on time in my experience.

In looking at the documentation for browsertime I noticed that there is a possibility of using ChromeDriver.exe instead of fully installing Chrome (more here).

Could you clarify what you mean about using chromedriver.exe instead of installing Chrome?
Looking at the browserstack link, one eventual goal related to this bug was to automate grabbing the latest (stable) chromedrivers from chrome-for-testing (CfT) dashboard but that would require our Windows Chrome to be reliably be up to date (like it is on linux/mac for the most part)

worth also mentioning, we unfortunately can't use the Chrome binaries from the CfT dashboard ^ as they are not built with PGO so we can't rely on those performance numbers.

After discussing on Slack, I will work on adding a startup script to check if the current version of chrome that's installed matches with what's available on https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json and if not, uninstall the current version and install the latest version. I will also add internal logging so we (release sre) can manage any failures that may impact the workers. All of this would happen prior to a test running, so optimizing for time+bandwidth are key.

Comment 14

[:jmoss]

We have successfully implemented updates to windows configuration code that pulls down the latest google chrome and installs it upon a new release being available using chocolatey and puppet.