Closed Bug 1632711 Opened 3 years ago Closed 2 years ago

Stand up Wordpress gutenberg against Firefox Nightly in CI


(Remote Protocol :: Agent, task, P2)



(Not tracked)



(Reporter: impossibus, Assigned: impossibus)


(Whiteboard: [puppeteer-beta2-mvp])

Set up gutenberg tests running against Firefox on Travis to check our progress more easily and for an eventual PR to

Whiteboard: [puppeteer-beta-reserve] → [puppeteer-beta2-mvp]
Summary: Stand up Wordpress gutenberg against Firefox Nightly in Travis CI → Stand up Wordpress gutenberg against Firefox Nightly in CI

The Gutenberg project uses GitHub Actions for their CI.

There's a Firefox Nightly job set up at

I will add a comment on this bug to document how to trigger such a job.

I will also add whimboo as repo owner.

Assignee: nobody → mjzffr

Going to and clicking on "Run workflow" will run Gutenberg e2e tests against the latest Firefox Nightly.

The default branch on mjzffr/gutenberg is firefox-ci for this purpose.

A few notes:

  • Gutenberg uses GitHub Actions for their CI jobs. I used largely the same configuration in the mjzffr/gutenberg fork.
  • Gutenberg e2e tests depend on Puppeteer 3. Upgrading a newer version of Puppeteer is non-trivial because other tools (like jest-puppeteer) depend specifically in Puppeteer <= 3. As far as Firefox support goes, the main difference between Puppeteer 3 and 5 is browser fetching and installation, as well as a few new default Firefox preferences. So the CI job uses mozdownload to fetch Firefox Nightly, and I set the missing Firefox preferences in the e2e test config.
Closed: 2 years ago
Resolution: --- → FIXED

Thank you Maja! When checking individual test jobs I can see a lot of test output, and I wonder if there is a dashboard or something else the Wordpress folks might use to keep an overview of failing/passing tests. The produced artifacts aren't that clear (at least not from the first check) in what's actually broken / failing. Or is there nothing and failures need to be investigated by observing the test output? That would be really tedious.

Flags: needinfo?(mjzffr)

Hi! :) I'm not aware of any dashboard used by the upstream gutenberg project to summarize CI output. I think they have pretty minimal output of just test summaries, whereas in the Mozilla fork I enabled browser trace logs. In any case, the best person to ask about their e2e tests and CI setup is gziolo on GitHub.

I think some extra work will be needed to green up the gutenberg tests again after changes to Remote Protocol since July 2020 -- that would make future test results easier to parse. The summary of setup steps on the project wiki is still valid

Also, I've resent the GitHub invite to you, Henrik.

Flags: needinfo?(mjzffr)
You need to log in before you can comment on or make changes to this bug.