1564256 - Expose browsertime to generic-worker based test tasks

Status: RESOLVED FIXED

(Testing :: Performance, task)

Description

[Nick Alexander :nalexander [he/him]]

The existing Raptor tasks run under mozharness, without a source checkout. That means the mach browsertime --setup command added in Bug 1543247 isn't a natural way to install the browsertime NPM modules. (And even if it was, a source checkout and mach command isn't a very efficient way to expose these things.)

This ticket tracks exposing browsertime to generic-worker based test tasks. The archive packaged will be mounted in generic-worker and needs to work as-is in that configuration. (I'm told that the Windows generic-worker configuration has Node.js available out of the box.)

I see several different options:

1. Produce a browsertime.tests.tar.gz tarball, like we do for raptor.tests.tar.gz. We might run ./mach browsertime --setup as part of the build for this, and then package it up. This isn't reasonable 'cuz we don't want to npm install from live networks during builds.

2. Build a toolchain task that runs mach browsertime --setup and then packages up tools/browsertime/node_modules in some form. There's no restriction on running mach commands in these tasks.

3. Build a fetch task that runs even closer to the metal: perhaps one that runs npm install directly rather than running mach browsertime --setup.

Right now I see no real difference between toolchain tasks and fetch tasks. I understand toolchain tasks more thoroughly, and the indexing seems closer to what we want, so I'll investigate that approach.

Comment 1

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1564256 - Part 1: Don't install optional browsertime packages in automation. r?brennie

browsertime depends on a few architecture and OS specific packages:

• sharp (libvips)
• geckodriver
• chromedriver

Our toolchain task packages up tools/browsertime/node_modules and
we'd like to use the resulting toolchain archive across all of our
test platforms. Since in automation we don't require sharp (which is
only used for screenshotting), and we provide geckodriver and
chromedriver at the task level, the simplest way is to make these
optionalDependencies at the NPM level and not install them in our
toolchain task.

Depends on D37132

Comment 2

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1564256 - Part 2: Produce browsertime.zip in a toolchain task. r?#firefox-build-system-reviewers

In browsertime.zip we should have:

browsertime/
package.json
package-lock.json
node_modules/
.bin/
browsertime -> ../browsertime/bin/browsertime.js
browsertime/
...

The idea is that we'll fetch browsertime.zip in a generic-worker
environment and be able to run Node.js from within the top level
browsertime/ directory.

Depends on D38772

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/187d7e74c0d2
Part 1: Don't install optional browsertime packages in automation. r=barret
https://hg.mozilla.org/integration/autoland/rev/bc060dfd5700
Part 2: Produce browsertime.zip in a toolchain task. r=mshal

Comment 4

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/187d7e74c0d2
https://hg.mozilla.org/mozilla-central/rev/bc060dfd5700