1646923 - puppeteer-expected.json doesn't list all the tests from the Puppeteer 3.1 upgrade

Status: RESOLVED FIXED

(Remote Protocol :: Agent, defect, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When I check the results of a puppeteer test job I can see:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306727659&repo=mozilla-central&lineNumber=48499

{"suites":153,"tests":654,"passes":350,"pending":54,"failures":250,"start ...

Which means there should be 654 tests in total here. Maja said we only skip 4 tests, so 650 should remain. Also 18 tests have multiple statuses, which should bring us to 632 tests as counted for https://github.com/puppeteer/ispuppeteerfirefoxready.

But if you look at the chart there are only ~575 results shown.

Each test in puppteer-expected.json spans three lines and with 1908 lines in total this are 636 listed tests. That means 18 tests don't seem to be listed. So with the update to Puppeteer 3.1 the expectation data from some newly added / removed tests are missing.

Tasks here:

• Ensure we have all the expected tests listed
• Why don't we fail when tests are missing in puppeteer-expected.json?
  When parsing the output we should fail if the number of total tests run doesn't match

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

The 18 unlisted tests seem to be input tests that are actually skipped.

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Why don't we fail when tests are missing in puppeteer-expected.json?

This is only if skipped tests are missing, due to https://searchfox.org/mozilla-central/rev/3d39d3b7dd1b2be30692d4541ea681614e34c786/remote/mach_commands.py#240-245

That also means that if a test is skipped, the puppeteer-expected.json lists some other status, no error is reported.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1646923 - [remote] Add metadata checks for skipped and extra tests

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1646923 - [remote] Update puppeteer metadata

Depends on D80796

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/df561c571cfd
[remote] Add metadata checks for skipped and extra tests r=remote-protocol-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/db74cdf9afe7
[remote] Update puppeteer metadata r=remote-protocol-reviewers,whimboo

Comment 6

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/df561c571cfd
https://hg.mozilla.org/mozilla-central/rev/db74cdf9afe7