1540655 - Create TaskCluster job for the Puppeteer test suite

Status: RESOLVED FIXED

(Remote Protocol :: Agent, enhancement, P1)

Description

[Andreas Tolfsen ❲:ato❳]

To track our continued progress towards complete Puppeteer support
we need to create a new Tier-2/Tier-3 TaskCluster job running the
Puppeteer test suite on try.

The Puppeteer test suite requires Node.js, but since this is now a
build-time dependency we should be fine in that area. My current
thinking is to vendor the test suite in central, then have a mach
command for pulling in new changes from upstream.

Comment 1

[Andreas Tolfsen ❲:ato❳]

I need to vendor the Puppeteer repository along with its dependencies
in central for this bug. My initial plan is to put the dependencies
under third_party/nodejs/ and annontate them with moz.yaml files,
and to put the Puppeteer repo under remote/test/puppeteer/.

I then intend to introduce a new mach command (something like
./mach remote vendor-puppeteer) to pull in the latest changes from
GitHub. This will allow the remote protocol to track progress towards
achieving full Puppeteer API support.

Does that all sound reasonable to you?

Comment 2

[James Graham [:jgraham]]

mach vendor already exists and seems like it should be the frontend here even if it requires a custom subcommand like mach vendor dav1d.

Comment 3

[Jason Laster [:jlast]]

That seems reasonable. What third party dependencies do you have?
We meet tuesdays at 12:30pm est in my vidyo room to discuss node mc topics if you'd like to discuss in more depth.

CC Dan Mosedale

Comment 4

[Alexandre Poirot [:ochameau]]

We may start with a forked version of puppeteer. For example, to workaround bug 1543115. But I imagine we will hit some other similar blockers. I can easily see us having to put some special bits to run the tests for firefox.
What is your plan to accomodate that?

Also, is there an urge to support mach vendor to update puppeteer?
Wouldn't it be easier to clone it manually and rather focus on getting the test running?
I imagine we could do that and still flag the files correctly as vendor via moz.yaml?

Comment 5

[Andreas Tolfsen ❲:ato❳]

I don’t think ./mach vendor supports Node.js quite yet, but my
thinking was to manage the third-party dependencies using that so
that they could live under third_party/.

As we need to lock the Puppeteer version to the remote agent and
have the ability to apply custom patches to it, I want to manually
export a git clone into remote/test/puppeteer/. I mentioned
./mach remote vendor-puppeteer because those steps can
be automated: not as a replacement for ./mach vendor.

Comment 6

[Julian Descottes [:jdescottes]]

For info I made a fork of puppeteer 1.6.2 that works with the current Firefox Nightly and doesn't rely on Juggler. It normally sets the correct preferences for the profile. I'm not sure we want to use exactly that version here, but at least it should work: https://github.com/juliandescottes/pptr-ff

Comment 7

[Andreas Tolfsen ❲:ato❳]

Yes, I did the same but with only the minimal amount of changes
required to make npm test run:
https://github.com/andreastt/puppeteer/tree/firefox

With my branch you still use output variables such as
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD and CHROME.

Since we arrived at the conclusion that we probably need to fork
Puppeteer regardless in order to add support for Firefox, going
with your fork long-term seems like the right thing to do.

Comment 8

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: build: sort MACH_MODULES;

Comment 9

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: build, remote: add mach command for vendoring Puppeteer;

Introduces "./mach remote vendor-puppeteer" for vendoring the
Puppeteer client without dependencies into remote/test/puppeteer/.

The particular checkout of Puppeteer is
https://github.com/andreastt/puppeteer/tree/firefox, which contains a
couple of hotfixes we need for the client to work with the Firefox
implementation of CDP.

The remote agent targets a specific version of Puppeteer, so it is
not suitable for this to be vendored under third_party/. We also
wouldn't want other code in central to accidentally use a patched fork.

The vendoring process is not part of "./mach vendor" because it does
not yet have Node.js support, and implementing that for mach is outside
the scope of getting the Puppeteer tests running with the remote agent.

Comment 10

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: vendor Puppeteer;

Puppeteer is licensed under the Apache-2.0 license.

No code from Puppeteer gets included in Firefox.

Comment 11

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: add mach command for running Puppeteer tests;

Introduces "./mach puppeteer-test" command for running the Puppeteer
tests against the remote agent. This has to be a top-level command
because the automatic test detection system in mach does not allow
us to delegate to a subcommand such as "./mach remote puppeteer-test".

The tests run against a fork of Puppeteer with hotfixes needed for
it to work with the CDP implementation in Firefox. This fork is
located at https://github.com/andreastt/puppeteer/tree/firefox, and
vendored under remote/test/puppeteer/ in a previous commit in this series.

Comment 12

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: ignore non-vendored Node.js assets;

We vendor the Puppeteer library, but not its dependencies.
When "npm install" is called in remote/test/puppeteer/, it puts
its dependencies under remote/test/puppeteer/node_modules/ and
generates a remote/test/puppeteer/package-lock.json file. We do
not want these to be checked in.

DONTBUILD

Comment 13

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: add ability to symlink Puppeteer;

Adds a --symlink <PATH> flag that instead of checkout the Puppeteer
source code from a git repo, allows one to symlink a local directory
into remote/test/puppeteer. This is useful for local development.

Comment 14

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: mozbuild: add puppeteer test flavor;

The Puppeteer test flavour are functional tests for the CDP-based
Puppeteer library from Google, that we want to run against our
implementation of CDP for Firefox.

They are distinct from the Firefox Puppeteer tests based on Marionette.

Comment 15

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: moztest: add test path resolution for Puppeteer tests;

Makes it possible for mach to resolve test paths for Puppeteer,
so that individual tests can be run from the command line using
"./mach test", as such:

% ./mach test remote/test/puppeteer/test/screenshot.spec.js

As the Puppeteer test suite is imported from upstream and we cannot
change this directory at will (i.e. to add test manifest files),
we take the same approach as for WPT and populate the manifest by
recursively walking the remote/test/puppeteer/test/**/*.spec.js file tree.

Comment 16

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: connect Puppeteer spec tests with mach test flavour;

This makes it possible to for mach to determine which test types
are likely to be relevant.

Comment 17

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: ci: add test job for Puppeteer tests;

This adds a "remote(pup)" TaskCluster job for running the Google
Puppeteer tests against the new Firefox remote agent.

Earlier in this patchset we added a "./mach puppeteer-test" command
that wraps npm, and this being a source-test with access to the
full checkout, invokes this command directly instead of going
through mozharness.

Comment 18

[Andreas Tolfsen ❲:ato❳]

try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=588eeea15278d70f7ae4ecaaff355803c58c8deb&selectedJob=254822601

It is expected to be failing on Tier-3 because we don’t support all the tests yet.

Comment 19

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: document how to run Puppeteer tests;

Comment 20

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1540655: remote: document how to vendor Puppeteer;

Comment 21

[Andreas Tolfsen ❲:ato❳]

try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=c92334de41fcf97a830edf649617c7365ef73641

remote(pup) is expected to fail.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/997d1568d944
build: sort MACH_MODULES; r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/b96dede008ad
build, remote: add mach command for vendoring Puppeteer; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/3b3a2a9fbc8b
remote: vendor Puppeteer; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/939ce2afcf0b
remote: add mach command for running Puppeteer tests; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/c0a80d37576d
remote: ignore non-vendored Node.js assets; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/8c67b494e207
mozbuild: add puppeteer test flavor; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/bb012df3018b
moztest: add test path resolution for Puppeteer tests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8d7bad30be46
remote: connect Puppeteer spec tests with mach test flavour; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/73236f81da44
ci: add test job for Puppeteer tests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/641a7cb25298
remote: document how to run Puppeteer tests; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/1a23d770d8a1
remote: document how to vendor Puppeteer; r=remote-protocol-reviewers,jdescottes

Comment 23

[Andrei Ciure[:aciure]]

Backed out 11 changesets (bug 1540655) for test_resolve.py perma failures

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7&selectedJob=261606119

failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=261606119&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7&searchStr=mb

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=261606138&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=1a23d770d8a1c0f0d13b5211c5b9380e69c000f7

backout: https://hg.mozilla.org/integration/autoland/rev/79127c2c712388688e6cbe5748b0d14d6efb766d

Comment 24

[Andreas Tolfsen ❲:ato❳]

This was a test failure in test_resolve.py caused by is_puppeteer_path()
trying to match on None values. I’ve rebased and fixed the test.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2b9955b0822
build: sort MACH_MODULES; r=firefox-build-system-reviewers,mshal
https://hg.mozilla.org/integration/autoland/rev/8f22cff02e35
build, remote: add mach command for vendoring Puppeteer; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/5b769a365120
remote: vendor Puppeteer; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/15c8a10a3994
remote: add mach command for running Puppeteer tests; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/beab1be0f328
remote: ignore non-vendored Node.js assets; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/4da1734351e6
mozbuild: add puppeteer test flavor; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/bf91c2f4f386
moztest: add test path resolution for Puppeteer tests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/23931313b016
remote: connect Puppeteer spec tests with mach test flavour; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/7712e6870bd3
ci: add test job for Puppeteer tests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/836e30244634
remote: document how to run Puppeteer tests; r=remote-protocol-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/8a9e9189cd98
remote: document how to vendor Puppeteer; r=remote-protocol-reviewers,jdescottes

Comment 26

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/f2b9955b0822
https://hg.mozilla.org/mozilla-central/rev/8f22cff02e35
https://hg.mozilla.org/mozilla-central/rev/5b769a365120
https://hg.mozilla.org/mozilla-central/rev/15c8a10a3994
https://hg.mozilla.org/mozilla-central/rev/beab1be0f328
https://hg.mozilla.org/mozilla-central/rev/4da1734351e6
https://hg.mozilla.org/mozilla-central/rev/bf91c2f4f386
https://hg.mozilla.org/mozilla-central/rev/23931313b016
https://hg.mozilla.org/mozilla-central/rev/7712e6870bd3
https://hg.mozilla.org/mozilla-central/rev/836e30244634
https://hg.mozilla.org/mozilla-central/rev/8a9e9189cd98