1913832 - [cdp] OOPIF pdf test incorrectly reported as unexpected missing

Status: RESOLVED FIXED

(Remote Protocol :: Agent, defect, P3)

Description

[Julian Descottes [:jdescottes]]

This is a recurring issue when doing puppeteer syncs, the test

[oopif.spec] OOPIF should evaluate on a page with a PDF viewer

has an explicit skip for Firefox + BiDi, and when running for cdp, our CI will then report

TEST-UNEXPECTED-MISSING Unexpected skipped OOPIF should evaluate on a page with a PDF viewer (oopif.spec.js)

However ALL oopifs tests should be skipped for Firefox+cdp per the following rule:

https://searchfox.org/mozilla-central/rev/e51b2630b44a8836a7ff35a876a2d8b555041d4a/remote/test/puppeteer/test/TestExpectations.json#586-592

{
  "testIdPattern": "[oopif.spec] *",
  "platforms": ["darwin", "linux", "win32"],
  "parameters": ["cdp", "firefox"],
  "expectations": ["SKIP"],
  "comment": "TODO: add a comment explaining why this expectation is required (include links to issues)"
},

I suspect this is a bug in our mach command around https://searchfox.org/mozilla-central/rev/e51b2630b44a8836a7ff35a876a2d8b555041d4a/remote/mach_commands.py#319-323

if status == "SKIP":
    self.logger.test_start(test_name)
    if self.expected and status not in expected:
        self.unexpected_skips.add(test_name)
    expected = ["SKIP"]

Comment 1

[Julian Descottes [:jdescottes]]

Attachment: Bug 1913832 - [puppeteer] Fix expectations filter for puppeteer-test command

Depends on D219565

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc3cdccae212
[puppeteer] Fix expectations filter for puppeteer-test command r=webdriver-reviewers,Sasha

Comment 3

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/dc3cdccae212