1700801 - [M-fis-xorig] TEST-UNEXPECTED-PASS | layout/generic/test/test_bug756984.html | fail-if condition in manifest - We expected at least one failure

Status: RESOLVED FIXED

(Core :: Layout, defect)

Description

[Chris Peterson [:cpeterson]]

ahal hit this TEST-UNEXPECTED-PASS when trying to to enable mochitest-plain in fission-xorigin mode on more platforms in bug 1694835:

https://treeherder.mozilla.org/jobs?repo=try&revision=a4813c46eff128482994870bf8a06677022cee24&selectedTaskRun=TjKj0jluRJyAFeNT5JSRbQ.0

TEST-UNEXPECTED-PASS | layout/generic/test/test_bug756984.html | fail-if condition in manifest - We expected at least one failure

This test was annotated to fail-if xorigin in bug 1602668:

https://searchfox.org/mozilla-central/rev/368607c4cd5be547021945e4ae60e8eb4365b3c4/layout/generic/test/mochitest.ini#81-83

[test_bug756984.html]
fail-if =
    os == 'linux' && os_version == '18.04'  # Bug 1600208 permafail on ubuntu1804
    xorigin

Why does the test fail for xorigin && !fission but unexpectedly pass for xorigin && fission? Sounds like a test bug. We should either fix the test, add a skip-if xorigin && fission, or always skip-if xorigin.

Comment 1

[Chris Peterson [:cpeterson]]

Patricia, is adding the [layout:triage-discuss] whiteboard tag still the correct method to get a bug on the Layout team's triage radar?

This layout test is annotated as failing in mochitest "xorigin" mode (by bug 1602668 with no explanation), but the test is unexpectedly passing in xorigin mode when Fission is enabled. Having a Layout engineer diagnose why the test fails in xorigin without Fission and unexpectedly passes with Fission would be a big help for the Fission team.

Comment 2

[Emilio Cobos Álvarez [:emilio]]

Do you have a log of the failure when fission is disabled?

Comment 3

[Chris Peterson [:cpeterson]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #2)

Do you have a log of the failure when fission is disabled?

Unfortunately, I don't.

These xorigin mode tests were originally intended to be run in "xorigin && !fission" and "xorigin && fission". The original annotations differentiated between those cases, but I see we currently only ever run these xorigin tests with both fission and xorigin mode enabled ("M-fis-xorig"). So this test's fail-if = xorigin annotation effectively implies and fission because we never test xorigin without fission. :(

Here is the mach command to run the test with xorigin without fission:

mach mochitest --enable-xorigin-tests

Comment 4

[Emilio Cobos Álvarez [:emilio]]

The test passes also without fission here, so we should just remove that fails-if annotation, looks like.

Comment 5

[Daniel Holbert [:dholbert]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)

The test passes also without fission here, so we should just remove that fails-if annotation, looks like.

Would you mind making that change?

Comment 6

[Emilio Cobos Álvarez [:emilio]]

Attachment: Bug 1700801 - Remove stale test annotation of test_bug756984 which passes with fission-xorigin. r=#layout-reviewers

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/977f76231d70
Remove stale test annotation of test_bug756984 which passes with fission-xorigin. r=layout-reviewers,TYLin

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/977f76231d70