Closed Bug 1646923 Opened 3 months ago Closed 3 months ago

puppeteer-expected.json doesn't list all the tests from the Puppeteer 3.1 upgrade

Categories

(Remote Protocol :: Agent, defect, P3)

Tracking

(firefox79 fixed)

RESOLVED FIXED
Firefox 79
Tracking Status
firefox79 --- fixed

People

(Reporter: whimboo, Assigned: maja_zf)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files)

When I check the results of a puppeteer test job I can see:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306727659&repo=mozilla-central&lineNumber=48499

{"suites":153,"tests":654,"passes":350,"pending":54,"failures":250,"start ...

Which means there should be 654 tests in total here. Maja said we only skip 4 tests, so 650 should remain. Also 18 tests have multiple statuses, which should bring us to 632 tests as counted for https://github.com/puppeteer/ispuppeteerfirefoxready.

But if you look at the chart there are only ~575 results shown.

Each test in puppteer-expected.json spans three lines and with 1908 lines in total this are 636 listed tests. That means 18 tests don't seem to be listed. So with the update to Puppeteer 3.1 the expectation data from some newly added / removed tests are missing.

Tasks here:

  • Ensure we have all the expected tests listed
  • Why don't we fail when tests are missing in puppeteer-expected.json?
    When parsing the output we should fail if the number of total tests run doesn't match
Assignee: nobody → mjzffr

The 18 unlisted tests seem to be input tests that are actually skipped.

Why don't we fail when tests are missing in puppeteer-expected.json?

This is only if skipped tests are missing, due to https://searchfox.org/mozilla-central/rev/3d39d3b7dd1b2be30692d4541ea681614e34c786/remote/mach_commands.py#240-245

That also means that if a test is skipped, the puppeteer-expected.json lists some other status, no error is reported.

Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/df561c571cfd
[remote] Add metadata checks for skipped and extra tests r=remote-protocol-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/db74cdf9afe7
[remote] Update puppeteer metadata r=remote-protocol-reviewers,whimboo
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 79
You need to log in before you can comment on or make changes to this bug.