Closed Bug 1528731 Opened 6 years ago Closed 5 years ago

Run tp6 against Google Chrome

Categories

(Testing :: Raptor, task, P3)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: davehunt, Unassigned)

References

Details

We're currently running the tp6 page load tests against Chromium. For a closer comparison between Firefox and Chrome we should run these against Google Chrome or Google Chrome Canary (pre-release).

Canary is not available on all platforms (Linux64, OSX, Win7 and Win10). When I originally looked into this, I was unable to find a method to automate the installation of Google Chrome proper, whereas Chromium was available as simple zip and not an installaer that needs to be driven manually.

However since then maybe things have changed - or if someone can figure out how to automatically install Google Chrome itself that would be awesome!

Assignee: nobody → igoldan
Priority: -- → P2

Eric: Do you have a preference of platforms/channels for us to support? If multiple, could you indicate which to prioritise?

Flags: needinfo?(esmyth)

:igoldan can you start by looking into using Canary on Windows? It looks like we may go for Canary wherever it's supported, and Dev on Linux.

Flags: needinfo?(igoldan)
Priority: P2 → P1

:davehunt, we haven't finalized the decision to go to Canary. We would still like to see tp6 results comparing Canary and Dev and to ensure that there are no unexpected issues with the proposal to change over to Canary on supported platforms.

Flags: needinfo?(esmyth)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #3)

:igoldan can you start by looking into using Canary on Windows? It looks like we may go for Canary wherever it's supported, and Dev on Linux.

I'll start the investigation.

Flags: needinfo?(igoldan)

(In reply to esmyth from comment #4)

:davehunt, we haven't finalized the decision to go to Canary. We would still like to see tp6 results comparing Canary and Dev and to ensure that there are no unexpected issues with the proposal to change over to Canary on supported platforms.

Thanks Eric. I haven't seen this request for comparing Canary and Dev, is this something you're waiting on my team on?

Flags: needinfo?(esmyth)

I started looking over this issue, but it seems things didn't change. I couldn't find simple Chrome binaries on any channel, only the installer for each platform.

Resuming the investigation, turns out that on Windows, Google Chrome is available as a custom, non-MSI installer. This complicates things even more, as msiexec could have helped us a lot in automating the installs.
Only Google Chrome Enterprise provides MSI installers...

So I found bug 1476372, which enabled Raptor to fetch & provision Chromium binaries for each platform.
Now my question is: how should we proceed with this bug? Should I leave Chromium tasks alone or should I simply replace them with proper Chrome tasks?

Personally, I would go with the 1st option: maintain current tasks which use Chromium and use bug 1476372 as reference for adding similar Chrome tasks. Once this bug is done & running fine, turn off Chromium tasks.

Dave, what do you think?

Flags: needinfo?(dave.hunt)
See Also: → 1476372

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #9)

So I found bug 1476372, which enabled Raptor to fetch & provision Chromium binaries for each platform.
Now my question is: how should we proceed with this bug? Should I leave Chromium tasks alone or should I simply replace them with proper Chrome tasks?

Personally, I would go with the 1st option: maintain current tasks which use Chromium and use bug 1476372 as reference for adding similar Chrome tasks. Once this bug is done & running fine, turn off Chromium tasks.

Dave, what do you think?

We don't currently intend to replace the Chromium jobs with Chrome jobs. We're still waiting for a final decision on the channels/platforms, but I believe the plan is to continue running against Chromium on a less frequent schedule (once per week, perhaps).

Flags: needinfo?(dave.hunt)

This sounds like I should go with the 1st option I mentioned, except turning off Chromium tasks.
Robert, I'm thinking of filing a separate bug just for replacing any current occurrence of Chrome with Chromium.
After I address this correction, I could just proceed with configuring the real Chrome tasks.

Does this sound ok to you?

Flags: needinfo?(rwood)
Flags: needinfo?(rwood)
Flags: needinfo?(rwood)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #11)

This sounds like I should go with the 1st option I mentioned, except turning off Chromium tasks.
Robert, I'm thinking of filing a separate bug just for replacing any current occurrence of Chrome with Chromium.
After I address this correction, I could just proceed with configuring the real Chrome tasks.

Does this sound ok to you?

+1 for making a new fetch task for Chrome proper, and adding new tasks that will use proper chrome (and leave existing that use Chromium).

filing a separate bug just for replacing any current occurrence of Chrome with Chromium.

Right, so you want to change all the existing raptor test names (i.e. 'raptor-tp6-amazon-chrome' to 'raptor-tp6-amazon-chromium') and then when you add new tests that use chrome you can use '*-chrome' again. This does make sense.

One thing, on the Raptor command line, we use '--app=chrome'. Instead of renaming that, I would add an additional option for '--app=chromium'. Reason is I when I run locally on Chrome, I am using Chrome proper, so I use '--app=chrome' all the time (and guessing others may be using that too, when running locally on Chrome proper).

Flags: needinfo?(rwood)

(In reply to Robert Wood [:rwood] from comment #12)

One thing, on the Raptor command line, we use '--app=chrome'. Instead of renaming that, I would add an additional option for '--app=chromium'. Reason is I when I run locally on Chrome, I am using Chrome proper, so I use '--app=chrome' all the time (and guessing others may be using that too, when running locally on Chrome proper).

Sure thing.

Depends on: 1539437

:davehunt, I did not file a request for the Dev vs Canary comparison. I mistakenly thought that Stuart was going to file one. I'll file a request shortly.

Flags: needinfo?(esmyth)

Peter, we need to literally install Google Chrome Canary on our physical Windows machines. We need to automate these installs & run them each day.
I know you have these dependency scripts with which you configure the installation of various tools.

What options do we have? For example, I'm thinking of having a cronjob which simply downloads the standalone installer on each physical machine on a specific filesystem location & either run the installer OR configure the dependency scripts to point to that installer.

I do have another question:

  • in your daily routines, if you push a new dependency to those json manifests, is it automatically picked by a background process? Or are you manually running that install command?
Flags: needinfo?(peter)

I'm the wrong Peter. I am just a user who joined bugzilla a week ago to file a bug on Thunderbird Lightning

Flags: needinfo?(peter)
Flags: needinfo?(pmoore)

After a lengthy discussion on IRC, I think the path to proceed is as follows:

  1. Linux

This is relatively straightforward - regular tasks can run with administrative privileges and install whatever they need to - the tasks run inside a docker container, and those are purged after the task completes, so the tasks cannot affect the host system installation. The assumption here is that it is ok for this to run inside a docker-container without impacting validity of performance results. If this is not the case, and we can't run these tasks in docker-worker, we probably should run using generic-worker native engine, similar to my suggestion for Mac below.

  1. Mac

We probably want a dedicated worker type, and run the tasks using generic-worker with the current native engine we have, with tasks running as the worker user. There is a risk that the Google Chrome installations might not leave the workers in a pristine state after the task completes, but we don't really have an option to run these tasks on virtualised hardware, or to reimage workers between task runs (as far as I am aware). However, this would be a good question for relops (:fubar).

  1. Windows

We probably want a similar setup to Windows talos workers - with a dedicated worker type that runs privileged tasks that have enough privileges to install Google Chrome. Again, like for Mac, there is a risk that tasks could leave machines in an altered state after the task completes, and I don't think, on hardware, we have an easy way to automatically reimage machines between task runs, so this might be a risk we can't easily avoid. Again, this is worth discussing with relops (:fubar).

Flags: needinfo?(pmoore)
Flags: needinfo?(klibby)

(In reply to Pete Moore [:pmoore][:pete] from comment #17)

After a lengthy discussion on IRC, I think the path to proceed is as follows:

  1. Linux

This is relatively straightforward - regular tasks can run with administrative privileges and install whatever they need to - the tasks run inside a docker container, and those are purged after the task completes, so the tasks cannot affect the host system installation. The assumption here is that it is ok for this to run inside a docker-container without impacting validity of performance results. If this is not the case, and we can't run these tasks in docker-worker, we probably should run using generic-worker native engine, similar to my suggestion for Mac below.

I'm pretty sure we need to run these perf tests on base metal & avoid any kind of virtualization.

As Peter noted on IRC: "we'll be generic-worker all the way then - probably you'll need a new worker type like releng-hardware/gecko-t-linux-talos".

(In reply to Pete Moore [:pmoore][:pete] from comment #17)

  1. Mac

We probably want a dedicated worker type, and run the tasks using generic-worker with the current native engine we have, with tasks running as the worker user. There is a risk that the Google Chrome installations might not leave the workers in a pristine state after the task completes, but we don't really have an option to run these tasks on virtualised hardware, or to reimage workers between task runs (as far as I am aware). However, this would be a good question for relops (:fubar).

I don't actually think we'll need to re image between task runs. It's redundant & too much resource intensive. Rather re image once per win/linux/osx machine each day.
Google Chrome Canary updates daily. So basically every day we'll reimage the bare metal machines then download the standalone installers then install them with admin rights.

(In reply to Pete Moore [:pmoore][:pete] from comment #17)

  1. Windows

We probably want a similar setup to Windows talos workers - with a dedicated worker type that runs privileged tasks that have enough privileges to install Google Chrome. Again, like for Mac, there is a risk that tasks could leave machines in an altered state after the task completes, and I don't think, on hardware, we have an easy way to automatically reimage machines between task runs, so this might be a risk we can't easily avoid. Again, this is worth discussing with relops (:fubar).

If we're not certain about the re imaging mechanism, then I don't think that simply running the installers is the best approach.

Another thing to consider: what if a new Google Chrome Canary version is simply broken? That would affect all Raptor-ChC() jobs, which are around 25 per platform. Even if Raptor would automatically request a re image, the problem will persist all day long.
Basically, the physical machines will be occupied with only installing/re imaging because of an external issue. Until a proper Chrome Canary version is released.

Given these risks, I'm inclined to abort this approach and look for another one. But I'm curious about Kendall's opinion. Maybe there's still some hope.

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #20)

I don't actually think we'll need to re image between task runs. It's redundant & too much resource intensive. Rather re image once per win/linux/osx machine each day.
Google Chrome Canary updates daily. So basically every day we'll reimage the bare metal machines then download the standalone installers then install them with admin rights.

Re-imaging daily isn't possible with macOS. Even weekly might be too much depending on how many workers we're looking at. Apple simply isn't interested in providing tooling to do this, and in 10.14 have moved even further away. Windows and Linux will be possible this quarter, but I'm still not sure that this is the best method to use.

What are the requirements? Why do we need to reimage at all, when we don't for any other tests?

Flags: needinfo?(klibby)

(In reply to Kendall Libby [:fubar] (he/him) from comment #23)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #20)

I don't actually think we'll need to re image between task runs. It's redundant & too much resource intensive. Rather re image once per win/linux/osx machine each day.
Google Chrome Canary updates daily. So basically every day we'll reimage the bare metal machines then download the standalone installers then install them with admin rights.

Re-imaging daily isn't possible with macOS. Even weekly might be too much depending on how many workers we're looking at. Apple simply isn't interested in providing tooling to do this, and in 10.14 have moved even further away. Windows and Linux will be possible this quarter, but I'm still not sure that this is the best method to use.

What are the requirements?

The requirements are these: we want to add somewhere around 25 Raptor tests on all win/macosx/linux platforms, which need to run on physical machines, without any kind of virtualization (e.g. docker). These tests will run on Google Chrome Canary on win & macosx and Google Chrome Dev on linux.

The first step to accomplish this is to provision these physical machines with the appropriate Chrome browser.

Why do we need to reimage at all, when we don't for any other tests?

Because we don't trust Google Chrome's installers. Google doesn't officially provide plain portable binaries for its browser, just standalone installers.
E.g. it's likely one of the Chrome Canary versions would have some issues while installing/uninstalling itself. This could potentially corrupt/alter the system state of the physical machines. Easy re imaging would ensure our machines can fallback to a pristine state, before any Chrome installers were run.

Dave, I believe this pretty much concludes that running the Chrome installer on the physical machines isn't an option for us.
Even if we'd be happy with the re imaging approach for Windows & Linux, we'll lose consistency, as we'll need a different approach for provisioning macosx machines. What's more: re imaging for Windows & Linux is still WIP, while our due date is short.

I also insisted on using just Linux for installing with admin privileges.

Peter mentioned 2 other options:

2: another option is that we don't install as part of a task, but install as part of the machine setup (on windows this would be in github.com/mozilla-relops/OpenCloudConfig)

3: another option is that we write something that can extract the installation from the msi, and install that instead, if at all possible, that doesn't require admin privileges

For option #2 from comment 26, I have some questions:

  1. Can this machine setup run daily?
  2. Does this setup have fallback mechanisms? E.g. if the installation process goes bad, is it reverted in a clean way, such that the system state doesn't get corrupted (junk registries on Windows, junk files on other OSes etc)?
  3. Can we provide URL paths to these installers? Or must the app/dependency be available on some Windows hosted app provider (Windows Store or something)?
Flags: needinfo?(pmoore)

This is another lengthy discussion on IRC still insisting on 1st approach. It started from the fact that maybe we could still install on Linux bare metal, using apt-get. Basically, consider apt-get as trustworthy for keeping the system state uncorrupted.
Turns out apt-get isn't trustworthy. We still need isolation mechanisms. Chroot is a solution to that, but it's uncertain whether it affects performance or not.

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #27)

For option #2 from comment 26, I have some questions:

  1. Can this machine setup run daily?
  2. Does this setup have fallback mechanisms? E.g. if the installation process goes bad, is it reverted in a clean way, such that the system state doesn't get corrupted (junk registries on Windows, junk files on other OSes etc)?
  3. Can we provide URL paths to these installers? Or must the app/dependency be available on some Windows hosted app provider (Windows Store or something)?

Can we back up here, please?

We're experienced at automating an environment for testing Firefox. Your fears about how Chrome might pollute a direct hardware install are quite valid and will need investigation before we proceed. Our hardware pools are already lean, so we can't really afford to be taking machines offline for the hours it might take to re-image them constantly. Segmenting the existing pool might not be possible depending on the capacity hit it would mean for Firefox testing.

In short, you're asking for a lot of new work from my team (Taskcluster) and fubar's team (relops). We've already committed to work for Q2, and trying to shoehorn new things in right after we've finalized our OKRs isn't cool.

Can we get Yan and Laura to agree on whether this is important enough to defer another goal that we've already committed to?

Flags: needinfo?(pmoore)

All, this sounds like way more work than we bargained for when this request was first brought up. Thanks for digging in to uncover these details.

For now, we will make do with manual comparisons against Chrome as needed. We should do this work eventually, but we'll schedule it in for some future quarter. I think there's also a larger discussion here around how can we more dynamically provision and update many kinds of software (like other browsers, but also av, firewalls, etc.) that may be useful for performance testing (either direct comparisons between browsers, or because of the potential impact to performance via the existence of some types of software). Those are bigger questions and warrant bigger discussions. Lets put a pin in this for now.

(In reply to Chris Cooper [:coop] pronoun: he from comment #29)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #27)

For option #2 from comment 26, I have some questions:

  1. Can this machine setup run daily?
  2. Does this setup have fallback mechanisms? E.g. if the installation process goes bad, is it reverted in a clean way, such that the system state doesn't get corrupted (junk registries on Windows, junk files on other OSes etc)?
  3. Can we provide URL paths to these installers? Or must the app/dependency be available on some Windows hosted app provider (Windows Store or something)?

Can we back up here, please?

We're experienced at automating an environment for testing Firefox. Your fears about how Chrome might pollute a direct hardware install are quite valid and will need investigation before we proceed. Our hardware pools are already lean, so we can't really afford to be taking machines offline for the hours it might take to re-image them constantly. Segmenting the existing pool might not be possible depending on the capacity hit it would mean for Firefox testing.

In short, you're asking for a lot of new work from my team (Taskcluster) and fubar's team (relops). We've already committed to work for Q2, and trying to shoehorn new things in right after we've finalized our OKRs isn't cool.

I definitely wouldn't want to propose new quarterly plans, as I am looking for the best approach using precisely the current options/features we have at our disposal.
Reminding the main idea behind 1st approach: install Chrome on bare machines with admin rights. Clearly, not suited for us. I agree with Chris on backing out on this. (Why bare metal? Because perf tests should only run that way.)

But we surely have other options that don't need extra prior work. Peter Moore mentioned that we can install Chrome without admin rights on our Windows physical machines. This is a variation to the 1st approach, literally: install Chrome on Windows bare metal without admin rights. I want to continue researching on this.

Basically, fetch Chrome, install it on a newly created guest account, run the Raptor test then destroy that account on exit with all its junk files/registries etc. That guest account acts as a trustworthy isolation mechanism. Thus, the admin-level system state doesn't get polluted.

This approach seems suited for Windows only. For Linux & OSX I'll look for a different approach, as there aren't any guest account mechanisms implemented for them yet.

(In reply to Stuart Philp :sphilp from comment #30)

I think there's also a larger discussion here around how can we more dynamically provision and update many kinds of software (like other browsers, but also av, firewalls, etc.) that may be useful for performance testing (either direct comparisons between browsers, or because of the potential impact to performance via the existence of some types of software). Those are bigger questions and warrant bigger discussions. Lets put a pin in this for now.

Agreed; we have a request for figuring out how to add 3rd party software for interaction testing (eg anti-virus), so extending that to include something like Chrome Canary/Dev could follow a similar path. The main plot twist is the bare metal side, but we've been working hard on having a MUCH better re-imaging story.

Priority: P1 → P3
Assignee: igoldan → nobody

Moving to to block bug 1563708, although I suspect we'll need a RelOps bug to handle provisioning Chrome, which will be a blocker. Waiting for further information on the meta bug for now.

Blocks: 1563708

:sparky can this be closed now?

Flags: needinfo?(gmierz2)
No longer blocks: 1563708
Depends on: 1563708
Type: enhancement → task

Yup! It's running on central now.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(gmierz2)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.