Closed Bug 1632711 Opened 5 years ago Closed 4 years ago

Stand up Wordpress gutenberg against Firefox Nightly in CI

Categories

(Remote Protocol :: Agent, task, P2)

Product:
Remote Protocol
Component:
Agent
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: impossibus, Assigned: impossibus)

Details

(Whiteboard: [puppeteer-beta2-mvp])

Set up gutenberg tests running against Firefox on Travis to check our progress more easily and for an eventual PR to https://github.com/WordPress/gutenberg

Whiteboard: [puppeteer-beta-reserve] → [puppeteer-beta2-mvp]
Summary: Stand up Wordpress gutenberg against Firefox Nightly in Travis CI → Stand up Wordpress gutenberg against Firefox Nightly in CI

The Gutenberg project uses GitHub Actions for their CI.

There's a Firefox Nightly job set up at https://github.com/mjzffr/gutenberg/tree/firefox-ci

I will add a comment on this bug to document how to trigger such a job.

I will also add whimboo as repo owner.

Assignee: nobody → mjzffr
Status: NEW → ASSIGNED

Going to https://github.com/mjzffr/gutenberg/actions?query=workflow%3A%22End-to-End+Tests%22 and clicking on "Run workflow" will run Gutenberg e2e tests against the latest Firefox Nightly.

The default branch on mjzffr/gutenberg is firefox-ci for this purpose.

A few notes:

  • Gutenberg uses GitHub Actions for their CI jobs. I used largely the same configuration in the mjzffr/gutenberg fork.
  • Gutenberg e2e tests depend on Puppeteer 3. Upgrading a newer version of Puppeteer is non-trivial because other tools (like jest-puppeteer) depend specifically in Puppeteer <= 3. As far as Firefox support goes, the main difference between Puppeteer 3 and 5 is browser fetching and installation, as well as a few new default Firefox preferences. So the CI job uses mozdownload to fetch Firefox Nightly, and I set the missing Firefox preferences in the e2e test config.
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED

Thank you Maja! When checking individual test jobs I can see a lot of test output, and I wonder if there is a dashboard or something else the Wordpress folks might use to keep an overview of failing/passing tests. The produced artifacts aren't that clear (at least not from the first check) in what's actually broken / failing. Or is there nothing and failures need to be investigated by observing the test output? That would be really tedious.

Flags: needinfo?(mjzffr)

Hi! :) I'm not aware of any dashboard used by the upstream gutenberg project to summarize CI output. I think they have pretty minimal output of just test summaries, whereas in the Mozilla fork I enabled browser trace logs. In any case, the best person to ask about their e2e tests and CI setup is gziolo on GitHub.

I think some extra work will be needed to green up the gutenberg tests again after changes to Remote Protocol since July 2020 -- that would make future test results easier to parse. The summary of setup steps on the project wiki is still valid https://wiki.mozilla.org/Remote/GutenbergCDPUsage

Also, I've resent the GitHub invite to you, Henrik.

Flags: needinfo?(mjzffr)
You need to log in before you can comment on or make changes to this bug.