[meta] Support running WordPress test suite against Firefox

NEW
Unassigned

Status

task
4 months ago
8 days ago

People

(Reporter: ochameau, Unassigned)

Tracking

(Depends on 12 bugs, Blocks 2 bugs, {meta})

Details

WordPress is using Puppeteer to run integration tests and for now only run them against chromium.

The test scripts are available on the Gutenberg repo:
https://github.com/WordPress/gutenberg/tree/master/packages/e2e-tests/specs
Gutenberg is a sub-component of WordPress dedicated to its editor component.

How to run the tests:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ npm install
$ vi node_modules/puppeteer/lib/Launcher.js

Replace the following line:

     const rl = readline.createInterface({ input: chromeProcess.stderr });

by:

     const rl = readline.createInterface({ input: chromeProcess.stdout });
$ vi packages/scripts/config/puppeteer.config.js

Add the following line, that refers to your local firefox build

   executablePath: "/path/to/mozilla-central/your-obj-dir/dist/bin/firefox",
$ ./bin/setup-local-env.sh # it requires some dependencies to be installed, like docker
$ npm run build
$ npm run test-e2e # this will run all the tests
$ npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js # this will run just one test

For extra debugging you may use DEBUG env variable like this:

$ DEBUG=puppeteer:session npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js
Depends on: 1539210
Depends on: 1539213
Depends on: 1539221

Here is an attempt to see what are the puppeteer APIs used by Gutenberg tests involving puppeteer:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ git grep -E "(page|browser|context)\." packages/e2e-tests packages/e2e-test-utils/ | grep -oE "(page|browser|context)\.[^( ]+\(" | sort | uniq
browser.pages(
page.$(
page.$$(
page.click(
page.$$eval(
page.$eval(
page.evaluate(
page.focus(
page.goto(
page.keyboard.down(
page.keyboard.press(
page.keyboard.type(
page.keyboard.up(
page.mouse.click(
page.mouse.down(
page.mouse.move(
page.mouse.up(
page.on(
page.once(
page.reload(
page.removeListener(
page.select(
page.setOfflineMode(
page.setRequestInterception(
page.setViewport(
page.type(
page.url(
page.waitFor(
page.waitForFunction(
page.waitForNavigation(
page.waitForSelector(
page.waitForXPath(
page.$x(

This list reflects Puppeteer API and not CDP ones.
And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

Priority: -- → P3
Summary: Support running wordpress test suite against Firefox → Support running WordPress test suite against Firefox
Depends on: 1543071
Type: enhancement → task
Keywords: meta
Priority: P3 → --
Summary: Support running WordPress test suite against Firefox → [meta] Support running WordPress test suite against Firefox
Depends on: 1543095
Depends on: 1543099
Depends on: 1543098
Depends on: 1543115
Depends on: 1543142
Depends on: 1543151
Depends on: 1543185
Depends on: 1544417

And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

I opened PR which tries to bump Puppeteer to the latest version (1.14.0). There are a few failing tests which need to be updated before we can proceed.

Depends on: 1544445
Depends on: 1545724
Depends on: 1545726

I did some good progress on the PR which updates Puppeteer to version 1.15.0 in Gutenberg repository. I hope we get it merged later this week.

Depends on: 1548098
Depends on: 1548102
Depends on: 1549384
Depends on: 1549782
Depends on: 1549785
Depends on: 1549786

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox? The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point. At the moment we run e2e tests for users with 2 types of roles admin and author. We could experiment with running different browsers to get better coverage.

(In reply to Grzegorz Ziółkowski from comment #5)

The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point.

There is a command line option to run Firefox headlessly. You don't need a different version, you can pass --headless to the command used to start any Firefox.

(In reply to Grzegorz Ziółkowski from comment #5)

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Thanks a lot for doing that!
I think that's all on your side for now, we need some more time to strengthen puppeteer implementation for firefox.
I got a few Gutenberg tests passing locally, but it still depends on various patches that require tests and review.

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox?

Not that I know off, it looks like everything is based on npm packages.
The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Thank you both for you explanations!

The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Let me know if you need some more details from me. So far, the most significant inconvenience is that you need to download Chromium and soon Firefox when installing these npm packages. This slows down everything and influences not only local workflows but also Continues Integration for jobs which don't care about e2e tests at all. It might be not a big issue for many projects, but for the WordPress project, it becomes an issue because of scale.

We still haven't merged Puppeteer upgrade PR because we can't identify what is the root cause of intermittent failures of tests which operate on keyboard interactions and text selection. In the long run, having a way to test it with Firefox would be a huge improvement. Just saying :)

Depends on: 1552324
Depends on: 1552325
Depends on: 1552326
Depends on: 1553102

In comment 1, I mentioned the puppeteer's methods being used by Gutenberg test suite.

Here is now the underlying CDP methods and events that the whole test suite is using:

$ DEBUG=puppeteer:session npm run test-e2e packages/e2e-tests/specs/ > gutenberg-logs 2>&1
$ cat gutenberg-logs  | grep -oE '"method":"[^"]+\.[^"]+"' | sed 's/"method":"//' | sed 's/"//' | sort | uniq
DOM.getBoxModel
DOM.getContentQuads
DOM.setFileInputFiles
Emulation.setDeviceMetricsOverride
Emulation.setTouchEmulationEnabled
Input.dispatchKeyEvent
Input.dispatchMouseEvent
Inspector.detached
Log.enable
Log.entryAdded
Network.continueInterceptedRequest
Network.dataReceived
Network.emulateNetworkConditions
Network.enable
Network.loadingFailed
Network.loadingFinished
Network.requestIntercepted
Network.requestServedFromCache
Network.requestWillBeSent
Network.resourceChangedPriority
Network.responseReceived
Network.setCacheDisabled
Network.setRequestInterception
Page.bringToFront
Page.domContentEventFired
Page.enable
Page.frameAttached
Page.frameClearedScheduledNavigation
Page.frameDetached
Page.frameNavigated
Page.frameResized
Page.frameScheduledNavigation
Page.frameStartedLoading
Page.frameStoppedLoading
Page.getFrameTree
Page.handleJavaScriptDialog
Page.javascriptDialogClosed
Page.javascriptDialogOpening
Page.lifecycleEvent
Page.loadEventFired
Page.navigate
Page.navigatedWithinDocument
Page.reload
Page.setLifecycleEventsEnabled
Page.windowOpen
Performance.enable
Runtime.callFunctionOn
Runtime.consoleAPICalled
Runtime.enable
Runtime.evaluate
Runtime.executionContextCreated
Runtime.executionContextDestroyed
Runtime.executionContextsCleared
Runtime.getProperties
Runtime.releaseObject
Security.enable
Security.securityStateChanged
Target.setAutoAttach

Most of them are already having a related bugzilla entry.
A significant part has already been implemented, at least partially, so that tests exceptations are fulfilled.
And a couple may be unnecessary. For example Network.* events may not all be used by puppeteer and are most likely not necessary for the Gutenberg test suite. At first sight, only Network.requestWillBeSent and Network.responseReceived were necessary for Gutenberg.

Alexandre, would it be possible to also get the count of each of those commands? Might be good to know.

I’ve taken the liberty to store these on
https://wiki.mozilla.org/Remote/GutenbergCDPUsage for posterity.
Compare with https://wiki.mozilla.org/Remote/PuppeteerCDPUsage for
the total Puppeteer CDP surface.

I filed tracking bugs for the missing events and methods:

Inspector.detached

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1553159 to have
this domain created first.

Network.dataReceived

https://bugzilla.mozilla.org/show_bug.cgi?id=1553146

Network.resourceChangedPriority

https://bugzilla.mozilla.org/show_bug.cgi?id=1553147

Page.frameClearedScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553148

Page.frameResized

https://bugzilla.mozilla.org/show_bug.cgi?id=1553151

Page.frameScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553150

Page.frameStartedLoading

https://bugzilla.mozilla.org/show_bug.cgi?id=1553149

Page.javascriptDialogClosed

https://bugzilla.mozilla.org/show_bug.cgi?id=1553153

Page.windowOpen

https://bugzilla.mozilla.org/show_bug.cgi?id=1553155

Security.securityStateChanged

https://bugzilla.mozilla.org/show_bug.cgi?id=1553156

Depends on: 1535102
Depends on: 1553847
Depends on: 1553849
Depends on: 1553854
Depends on: 1561401
Depends on: 1562740
Depends on: 1563206
Depends on: 1563692
You need to log in before you can comment on or make changes to this bug.