Open Bug 1539202 (puppeteer-gutenberg) Opened 1 year ago Updated 12 days ago

[meta] Support running WordPress Gutenberg test suite against Firefox

Categories

(Remote Protocol :: Agent, task)

task
Not set
normal

Tracking

(Not tracked)

People

(Reporter: ochameau, Unassigned)

References

(Depends on 23 open bugs, Blocks 1 open bug, )

Details

(Keywords: meta)

User Story

See https://wiki.mozilla.org/Remote/GutenbergCDPUsage#Running_the_gutenberg_tests_against_Firefox

WordPress is using Puppeteer to run integration tests and for now only run them against chromium.

The test scripts are available on the Gutenberg repo:
https://github.com/WordPress/gutenberg/tree/master/packages/e2e-tests/specs
Gutenberg is a sub-component of WordPress dedicated to its editor component.

How to run the tests:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ npm install
$ vi node_modules/puppeteer/lib/Launcher.js

Replace the following line:

     const rl = readline.createInterface({ input: chromeProcess.stderr });

by:

     const rl = readline.createInterface({ input: chromeProcess.stdout });
$ vi packages/scripts/config/puppeteer.config.js

Add the following line, that refers to your local firefox build

   executablePath: "/path/to/mozilla-central/your-obj-dir/dist/bin/firefox",
$ ./bin/setup-local-env.sh # it requires some dependencies to be installed, like docker
$ npm run build
$ npm run test-e2e # this will run all the tests
$ npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js # this will run just one test

For extra debugging you may use DEBUG env variable like this:

$ DEBUG=puppeteer:session npx wp-scripts test-e2e --config packages/e2e-tests/jest.config.js packages/e2e-tests/specs/links.test.js
Depends on: 1539210
Depends on: 1539213
Depends on: 1539221

Here is an attempt to see what are the puppeteer APIs used by Gutenberg tests involving puppeteer:

$ git clone https://github.com/WordPress/gutenberg.git
$ cd gutenberg
$ git grep -E "(page|browser|context)\." packages/e2e-tests packages/e2e-test-utils/ | grep -oE "(page|browser|context)\.[^( ]+\(" | sort | uniq
browser.pages(
page.$(
page.$$(
page.click(
page.$$eval(
page.$eval(
page.evaluate(
page.focus(
page.goto(
page.keyboard.down(
page.keyboard.press(
page.keyboard.type(
page.keyboard.up(
page.mouse.click(
page.mouse.down(
page.mouse.move(
page.mouse.up(
page.on(
page.once(
page.reload(
page.removeListener(
page.select(
page.setOfflineMode(
page.setRequestInterception(
page.setViewport(
page.type(
page.url(
page.waitFor(
page.waitForFunction(
page.waitForNavigation(
page.waitForSelector(
page.waitForXPath(
page.$x(

This list reflects Puppeteer API and not CDP ones.
And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

Priority: -- → P3
Summary: Support running wordpress test suite against Firefox → Support running WordPress test suite against Firefox
Depends on: 1543071
Type: enhancement → task
Keywords: meta
Priority: P3 → --
Summary: Support running WordPress test suite against Firefox → [meta] Support running WordPress test suite against Firefox
Depends on: 1543095
Depends on: 1543099
Depends on: 1543098
Depends on: 1543115
Depends on: 1543142
Depends on: 1543151
Depends on: 1543185
Depends on: 1544417

And they are using Puppeteer version 1.6, which requires some low level CDP features like bug 1539210.

I opened PR which tries to bump Puppeteer to the latest version (1.14.0). There are a few failing tests which need to be updated before we can proceed.

Depends on: 1544445
Depends on: 1545724
Depends on: 1545726

I did some good progress on the PR which updates Puppeteer to version 1.15.0 in Gutenberg repository. I hope we get it merged later this week.

Depends on: 1548098
Depends on: 1548102
Depends on: 1549384
Depends on: 1549782
Depends on: 1549785
Depends on: 1549786

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox? The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point. At the moment we run e2e tests for users with 2 types of roles admin and author. We could experiment with running different browsers to get better coverage.

(In reply to Grzegorz Ziółkowski from comment #5)

The related question is if there is also a headless version of Firefox so we could enable it on Travis at some point.

There is a command line option to run Firefox headlessly. You don't need a different version, you can pass --headless to the command used to start any Firefox.

(In reply to Grzegorz Ziółkowski from comment #5)

https://github.com/WordPress/gutenberg/pull/14986 needs review and should be ready to merge. We will be using Puppeteer 1.15.0. What else do you need on our end?

Thanks a lot for doing that!
I think that's all on your side for now, we need some more time to strengthen puppeteer implementation for firefox.
I got a few Gutenberg tests passing locally, but it still depends on various patches that require tests and review.

Do you know if there any plans to defer installing Chromium until you run Puppeteer for the first time? Well, my point is why would you install Chromium if you want to test with Firefox?

Not that I know off, it looks like everything is based on npm packages.
The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Thank you both for you explanations!

The recent work on Jest seems to force the install of both chromium and firefox puppeteers:
https://github.com/smooth-code/jest-puppeteer/pull/220

I'll try to open up the discussion with puppeteers contributors.

Let me know if you need some more details from me. So far, the most significant inconvenience is that you need to download Chromium and soon Firefox when installing these npm packages. This slows down everything and influences not only local workflows but also Continues Integration for jobs which don't care about e2e tests at all. It might be not a big issue for many projects, but for the WordPress project, it becomes an issue because of scale.

We still haven't merged Puppeteer upgrade PR because we can't identify what is the root cause of intermittent failures of tests which operate on keyboard interactions and text selection. In the long run, having a way to test it with Firefox would be a huge improvement. Just saying :)

Depends on: 1552324
Depends on: 1552325
Depends on: 1552326
Depends on: 1553102

In comment 1, I mentioned the puppeteer's methods being used by Gutenberg test suite.

Here is now the underlying CDP methods and events that the whole test suite is using:

$ DEBUG=puppeteer:session npm run test-e2e packages/e2e-tests/specs/ > gutenberg-logs 2>&1
$ cat gutenberg-logs  | grep -oE '"method":"[^"]+\.[^"]+"' | sed 's/"method":"//' | sed 's/"//' | sort | uniq
DOM.getBoxModel
DOM.getContentQuads
DOM.setFileInputFiles
Emulation.setDeviceMetricsOverride
Emulation.setTouchEmulationEnabled
Input.dispatchKeyEvent
Input.dispatchMouseEvent
Inspector.detached
Log.enable
Log.entryAdded
Network.continueInterceptedRequest
Network.dataReceived
Network.emulateNetworkConditions
Network.enable
Network.loadingFailed
Network.loadingFinished
Network.requestIntercepted
Network.requestServedFromCache
Network.requestWillBeSent
Network.resourceChangedPriority
Network.responseReceived
Network.setCacheDisabled
Network.setRequestInterception
Page.bringToFront
Page.domContentEventFired
Page.enable
Page.frameAttached
Page.frameClearedScheduledNavigation
Page.frameDetached
Page.frameNavigated
Page.frameResized
Page.frameScheduledNavigation
Page.frameStartedLoading
Page.frameStoppedLoading
Page.getFrameTree
Page.handleJavaScriptDialog
Page.javascriptDialogClosed
Page.javascriptDialogOpening
Page.lifecycleEvent
Page.loadEventFired
Page.navigate
Page.navigatedWithinDocument
Page.reload
Page.setLifecycleEventsEnabled
Page.windowOpen
Performance.enable
Runtime.callFunctionOn
Runtime.consoleAPICalled
Runtime.enable
Runtime.evaluate
Runtime.executionContextCreated
Runtime.executionContextDestroyed
Runtime.executionContextsCleared
Runtime.getProperties
Runtime.releaseObject
Security.enable
Security.securityStateChanged
Target.setAutoAttach

Most of them are already having a related bugzilla entry.
A significant part has already been implemented, at least partially, so that tests exceptations are fulfilled.
And a couple may be unnecessary. For example Network.* events may not all be used by puppeteer and are most likely not necessary for the Gutenberg test suite. At first sight, only Network.requestWillBeSent and Network.responseReceived were necessary for Gutenberg.

Alexandre, would it be possible to also get the count of each of those commands? Might be good to know.

I’ve taken the liberty to store these on
https://wiki.mozilla.org/Remote/GutenbergCDPUsage for posterity.
Compare with https://wiki.mozilla.org/Remote/PuppeteerCDPUsage for
the total Puppeteer CDP surface.

I filed tracking bugs for the missing events and methods:

Inspector.detached

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1553159 to have
this domain created first.

Network.dataReceived

https://bugzilla.mozilla.org/show_bug.cgi?id=1553146

Network.resourceChangedPriority

https://bugzilla.mozilla.org/show_bug.cgi?id=1553147

Page.frameClearedScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553148

Page.frameResized

https://bugzilla.mozilla.org/show_bug.cgi?id=1553151

Page.frameScheduledNavigation

https://bugzilla.mozilla.org/show_bug.cgi?id=1553150

Page.frameStartedLoading

https://bugzilla.mozilla.org/show_bug.cgi?id=1553149

Page.javascriptDialogClosed

https://bugzilla.mozilla.org/show_bug.cgi?id=1553153

Page.windowOpen

https://bugzilla.mozilla.org/show_bug.cgi?id=1553155

Security.securityStateChanged

https://bugzilla.mozilla.org/show_bug.cgi?id=1553156

Depends on: 1535102
Depends on: 1553847
Depends on: 1553849
Depends on: 1553854
Depends on: 1561401
Depends on: 1562740
Depends on: 1563206
Depends on: 1564832
Depends on: 1563692
Depends on: 1567436
Depends on: 1567476

We have just merged https://github.com/WordPress/gutenberg/pull/16875 into master which upgrades Puppeteer to the latest version (1.19.0). I hope this will make testing Gutenberg with Firefox much easier :)

(In reply to Grzegorz Ziółkowski from comment #12)

We have just merged
https://github.com/WordPress/gutenberg/pull/16875 into master
which upgrades Puppeteer to the latest version (1.19.0). I hope
this will make testing Gutenberg with Firefox much easier :)

Great work, thank you!

A couple of notes about running these against Firefox today:

  • Add profile with relevant preferences set (like remote.enabled) to puppeteer.config.js with args: ['--profile', 'path/to/profile']
  • DEBUG=puppeteer:session doesn't work anymore, you want DEBUG=puppeteer:*
Summary: [meta] Support running WordPress test suite against Firefox → [meta] Support running WordPress Gutenberg test suite against Firefox
Alias: puppeteer-gutenberg
No longer depends on: 1552326
No longer depends on: 1552324
No longer depends on: 1549508
No longer depends on: 1549443
No longer depends on: 1553854
No longer depends on: 1549509
No longer depends on: 1549451
No longer depends on: 1587859
No longer depends on: 1587857
No longer depends on: 1549506
No longer depends on: 1590201
No longer depends on: 1549505
No longer depends on: 1589784
Depends on: 1593431
Depends on: 1591927
No longer depends on: 1593431
No longer depends on: 1564832
No longer depends on: 1549517
No longer depends on: 1549483
No longer depends on: 1553149
No longer depends on: 1591979
No longer depends on: 1549512
No longer depends on: 1549513
No longer depends on: 1549438
No longer depends on: 1587422
No longer depends on: 1549426

As Maja mentioned yesterday with bug 1544417 fixed we finally pass the setup steps, and can successfully run tests now. She will have a look in how many of them we pass, or fail.

Flags: needinfo?(mjzffr)

Most of the tests hang due to a "Stay on Page"/"Leave Page" dialog. ("This page is asking you to confirm that you want to leave..."). How do we handle those in other automation? Is there a relevant pref?

Due to those hangs, I don't have numbers from a full test run, but I can see lots of test steps succeeding both in headless and non-headless mode.

The other issue I see a lot is calls to the missing DOM.describeNode method. I'll add info on that to a more specific bug.

Flags: needinfo?(mjzffr)

Hm, sounds like bug 1545724 didn't fix the beforeunload prompt handling. Lets file a new bug for that.

Regarding DOM.describeNode I'm working on that right now. See the updates for it on bug 1607560. It's actually not a trivial one.

These are the changes to gutenberg test setup that make it work with Firefox: https://github.com/mjzffr/gutenberg/commit/b6b78af647deaed09b96b9d75e328db87b222350 -- essentially avoiding waitForSelector calls which depend on describeNode.

Maja, could you please also have a look at all the meta bugs which are in the list of task dependencies? I assume a couple of those we can replace with the real implementation bugs and lower the list of remaining dependencies a lot.

Flags: needinfo?(mjzffr)

(In reply to Maja Frydrychowicz :maja_zf (UTC-4) (maja@mozilla.com) from comment #18)

These are the changes to gutenberg test setup that make it work with Firefox: https://github.com/mjzffr/gutenberg/commit/b6b78af647deaed09b96b9d75e328db87b222350 -- essentially avoiding waitForSelector calls which depend on describeNode.

The above changes are no longer necessary.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #19)

Maja, could you please also have a look at all the meta bugs which are in the list of task dependencies? I assume a couple of those we can replace with the real implementation bugs and lower the list of remaining dependencies a lot.

I had a look and all the meta bugs still make sense to me.

Flags: needinfo?(mjzffr)

Ok, I just wondering what's missing for eg Runtime.callFunctionOn?

You need to log in before you can comment on or make changes to this bug.