Closed Bug 1646923 Opened 11 months ago Closed 11 months ago

puppeteer-expected.json doesn't list all the tests from the Puppeteer 3.1 upgrade


(Remote Protocol :: Agent, defect, P3)



(firefox79 fixed)

Firefox 79
Tracking Status
firefox79 --- fixed


(Reporter: whimboo, Assigned: impossibus)




(Keywords: regression)


(2 files)

When I check the results of a puppeteer test job I can see:

{"suites":153,"tests":654,"passes":350,"pending":54,"failures":250,"start ...

Which means there should be 654 tests in total here. Maja said we only skip 4 tests, so 650 should remain. Also 18 tests have multiple statuses, which should bring us to 632 tests as counted for

But if you look at the chart there are only ~575 results shown.

Each test in puppteer-expected.json spans three lines and with 1908 lines in total this are 636 listed tests. That means 18 tests don't seem to be listed. So with the update to Puppeteer 3.1 the expectation data from some newly added / removed tests are missing.

Tasks here:

  • Ensure we have all the expected tests listed
  • Why don't we fail when tests are missing in puppeteer-expected.json?
    When parsing the output we should fail if the number of total tests run doesn't match
Assignee: nobody → mjzffr

The 18 unlisted tests seem to be input tests that are actually skipped.

Why don't we fail when tests are missing in puppeteer-expected.json?

This is only if skipped tests are missing, due to

That also means that if a test is skipped, the puppeteer-expected.json lists some other status, no error is reported.

Pushed by
[remote] Add metadata checks for skipped and extra tests r=remote-protocol-reviewers,whimboo
[remote] Update puppeteer metadata r=remote-protocol-reviewers,whimboo
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 79
You need to log in before you can comment on or make changes to this bug.