1539202 - (puppeteer-gutenberg) [meta] Support running WordPress Gutenberg test suite against Firefox

Status: RESOLVED INACTIVE

(Remote Protocol :: Agent, task)

Description

[Alexandre Poirot [:ochameau]]

WordPress is using Puppeteer to run integration tests and for now only run them against chromium.

The test scripts are available on the Gutenberg repo:
https://github.com/WordPress/gutenberg/tree/master/packages/e2e-tests/specs
Gutenberg is a sub-component of WordPress dedicated to its editor component.

How to run the tests:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ npm install
$ vi node_modules/puppeteer/lib/Launcher.js

Replace the following line:

     const rl = readline.createInterface({ input: chromeProcess.stderr });

by:

     const rl = readline.createInterface({ input: chromeProcess.stdout });

$ vi packages/scripts/config/puppeteer.config.js

Add the following line, that refers to your local firefox build

   executablePath: "/path/to/mozilla-central/your-obj-dir/dist/bin/firefox",

$ ./bin/setup-local-env.sh # it requires some dependencies to be installed, like docker
$ npm run build
$ npm run test-e2e # this will run all the tests
$ npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js # this will run just one test

For extra debugging you may use DEBUG env variable like this:

$ DEBUG=puppeteer:session npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js

Comment 1

[Alexandre Poirot [:ochameau]]

Here is an attempt to see what are the puppeteer APIs used by Gutenberg tests involving puppeteer:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ git grep -E "(page|browser|context)\." packages/e2e-tests packages/e2e-test-utils/ | grep -oE "(page|browser|context)\.[^( ]+\(" | sort | uniq
browser.pages(
page.$(
page.$$(
page.click(
page.$$eval(
page.$eval(
page.evaluate(
page.focus(
page.goto(
page.keyboard.down(
page.keyboard.press(
page.keyboard.type(
page.keyboard.up(
page.mouse.click(
page.mouse.down(
page.mouse.move(
page.mouse.up(
page.on(
page.once(
page.reload(
page.removeListener(
page.select(
page.setOfflineMode(
page.setRequestInterception(
page.setViewport(
page.type(
page.url(
page.waitFor(
page.waitForFunction(
page.waitForNavigation(
page.waitForSelector(
page.waitForXPath(
page.$x(

This list reflects Puppeteer API and not CDP ones.
And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

Comment 2

[Grzegorz Ziółkowski]

And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

I opened PR which tries to bump Puppeteer to the latest version (1.14.0). There are a few failing tests which need to be updated before we can proceed.

Comment 3

[Yulia Startsev [:yulia]]

Linking the PR here for context: https://github.com/WordPress/gutenberg/pull/14986

Comment 4

[Grzegorz Ziółkowski]

I did some good progress on the PR which updates Puppeteer to version 1.15.0 in Gutenberg repository. I hope we get it merged later this week.

Comment 5

[Grzegorz Ziółkowski]

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox? The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point. At the moment we run e2e tests for users with 2 types of roles admin and author. We could experiment with running different browsers to get better coverage.

Comment 6

[Brian Grinstead [:bgrins]]

(In reply to Grzegorz Ziółkowski from comment #5)

The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point.

There is a command line option to run Firefox headlessly. You don't need a different version, you can pass --headless to the command used to start any Firefox.

Comment 7

[Alexandre Poirot [:ochameau]]

(In reply to Grzegorz Ziółkowski from comment #5)

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Thanks a lot for doing that!
I think that's all on your side for now, we need some more time to strengthen puppeteer implementation for firefox.
I got a few Gutenberg tests passing locally, but it still depends on various patches that require tests and review.

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox?

Not that I know off, it looks like everything is based on npm packages.
The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Comment 8

[Grzegorz Ziółkowski]

Thank you both for you explanations!

The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Let me know if you need some more details from me. So far, the most significant inconvenience is that you need to download Chromium and soon Firefox when installing these npm packages. This slows down everything and influences not only local workflows but also Continues Integration for jobs which don't care about e2e tests at all. It might be not a big issue for many projects, but for the WordPress project, it becomes an issue because of scale.

We still haven't merged Puppeteer upgrade PR because we can't identify what is the root cause of intermittent failures of tests which operate on keyboard interactions and text selection. In the long run, having a way to test it with Firefox would be a huge improvement. Just saying :)

Comment 9

[Alexandre Poirot [:ochameau]]

In comment 1, I mentioned the puppeteer's methods being used by Gutenberg test suite.

Here is now the underlying CDP methods and events that the whole test suite is using:

$ DEBUG=puppeteer:session npm run test-e2e packages/e2e-tests/specs/ > gutenberg-logs 2>&1
$ cat gutenberg-logs  | grep -oE '"method":"[^"]+\.[^"]+"' | sed 's/"method":"//' | sed 's/"//' | sort | uniq
DOM.getBoxModel
DOM.getContentQuads
DOM.setFileInputFiles
Emulation.setDeviceMetricsOverride
Emulation.setTouchEmulationEnabled
Input.dispatchKeyEvent
Input.dispatchMouseEvent
Inspector.detached
Log.enable
Log.entryAdded
Network.continueInterceptedRequest
Network.dataReceived
Network.emulateNetworkConditions
Network.enable
Network.loadingFailed
Network.loadingFinished
Network.requestIntercepted
Network.requestServedFromCache
Network.requestWillBeSent
Network.resourceChangedPriority
Network.responseReceived
Network.setCacheDisabled
Network.setRequestInterception
Page.bringToFront
Page.domContentEventFired
Page.enable
Page.frameAttached
Page.frameClearedScheduledNavigation
Page.frameDetached
Page.frameNavigated
Page.frameResized
Page.frameScheduledNavigation
Page.frameStartedLoading
Page.frameStoppedLoading
Page.getFrameTree
Page.handleJavaScriptDialog
Page.javascriptDialogClosed
Page.javascriptDialogOpening
Page.lifecycleEvent
Page.loadEventFired
Page.navigate
Page.navigatedWithinDocument
Page.reload
Page.setLifecycleEventsEnabled
Page.windowOpen
Performance.enable
Runtime.callFunctionOn
Runtime.consoleAPICalled
Runtime.enable
Runtime.evaluate
Runtime.executionContextCreated
Runtime.executionContextDestroyed
Runtime.executionContextsCleared
Runtime.getProperties
Runtime.releaseObject
Security.enable
Security.securityStateChanged
Target.setAutoAttach

Most of them are already having a related bugzilla entry.
A significant part has already been implemented, at least partially, so that tests exceptations are fulfilled.
And a couple may be unnecessary. For example Network.* events may not all be used by puppeteer and are most likely not necessary for the Gutenberg test suite. At first sight, only Network.requestWillBeSent and Network.responseReceived were necessary for Gutenberg.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Alexandre, would it be possible to also get the count of each of those commands? Might be good to know.

Comment 11

[Andreas Tolfsen ❲:ato❳]

I’ve taken the liberty to store these on
https://wiki.mozilla.org/Remote/GutenbergCDPUsage for posterity.
Compare with https://wiki.mozilla.org/Remote/PuppeteerCDPUsage for
the total Puppeteer CDP surface.

I filed tracking bugs for the missing events and methods:

Inspector.detached

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1553159 to have
this domain created first.

Network.dataReceived

https://bugzilla.mozilla.org/show_bug.cgi?id=1553146

Network.resourceChangedPriority

https://bugzilla.mozilla.org/show_bug.cgi?id=1553147

Page.frameClearedScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553148

Page.frameResized

https://bugzilla.mozilla.org/show_bug.cgi?id=1553151

Page.frameScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553150

Page.frameStartedLoading

https://bugzilla.mozilla.org/show_bug.cgi?id=1553149

Page.javascriptDialogClosed

https://bugzilla.mozilla.org/show_bug.cgi?id=1553153

Page.windowOpen

https://bugzilla.mozilla.org/show_bug.cgi?id=1553155

Security.securityStateChanged

https://bugzilla.mozilla.org/show_bug.cgi?id=1553156

Comment 12

[Grzegorz Ziółkowski]

We have just merged https://github.com/WordPress/gutenberg/pull/16875 into master which upgrades Puppeteer to the latest version (1.19.0). I hope this will make testing Gutenberg with Firefox much easier :)

Comment 13

[Andreas Tolfsen ❲:ato❳]

(In reply to Grzegorz Ziółkowski from comment #12)

We have just merged
https://github.com/WordPress/gutenberg/pull/16875 into master
which upgrades Puppeteer to the latest version (1.19.0). I hope
this will make testing Gutenberg with Firefox much easier :)

Great work, thank you!

Comment 14

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

A couple of notes about running these against Firefox today:

• Add profile with relevant preferences set (like remote.enabled) to puppeteer.config.js with args: ['--profile', 'path/to/profile']
• DEBUG=puppeteer:session doesn't work anymore, you want DEBUG=puppeteer:*

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

As Maja mentioned yesterday with bug 1544417 fixed we finally pass the setup steps, and can successfully run tests now. She will have a look in how many of them we pass, or fail.

Comment 16

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Most of the tests hang due to a "Stay on Page"/"Leave Page" dialog. ("This page is asking you to confirm that you want to leave..."). How do we handle those in other automation? Is there a relevant pref?

Due to those hangs, I don't have numbers from a full test run, but I can see lots of test steps succeeding both in headless and non-headless mode.

The other issue I see a lot is calls to the missing DOM.describeNode method. I'll add info on that to a more specific bug.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Hm, sounds like bug 1545724 didn't fix the beforeunload prompt handling. Lets file a new bug for that.

Regarding DOM.describeNode I'm working on that right now. See the updates for it on bug 1607560. It's actually not a trivial one.

Comment 18

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

These are the changes to gutenberg test setup that make it work with Firefox: https://github.com/mjzffr/gutenberg/commit/b6b78af647deaed09b96b9d75e328db87b222350 -- essentially avoiding waitForSelector calls which depend on describeNode.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Maja, could you please also have a look at all the meta bugs which are in the list of task dependencies? I assume a couple of those we can replace with the real implementation bugs and lower the list of remaining dependencies a lot.

Comment 20

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(In reply to Maja Frydrychowicz :maja_zf (UTC-4) ([e-mail]) from comment #18)

These are the changes to gutenberg test setup that make it work with Firefox: https://github.com/mjzffr/gutenberg/commit/b6b78af647deaed09b96b9d75e328db87b222350 -- essentially avoiding waitForSelector calls which depend on describeNode.

The above changes are no longer necessary.

Comment 21

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #19)

Maja, could you please also have a look at all the meta bugs which are in the list of task dependencies? I assume a couple of those we can replace with the real implementation bugs and lower the list of remaining dependencies a lot.

I had a look and all the meta bugs still make sense to me.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Ok, I just wondering what's missing for eg Runtime.callFunctionOn?

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

It's not on our current list of goals. Closing as inactive.