Closed Bug 1511915 Opened 5 years ago Closed 5 years ago

Perma Fail Tier 2 Android /css/css-shapes/shape-outside/values/shape-margin-001.html | XXXX - computed - assert_equals: expected "XXXpx" but got "XXXpx"

Categories

(Core :: CSS Parsing and Computation, defect, P5)

defect

Tracking

()

RESOLVED FIXED
mozilla66
Tracking Status
firefox66 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: botond)

References

Details

Attachments

(1 file)

Filed by: btara [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=215346752&repo=mozilla-central

https://queue.taskcluster.net/v1/task/bb5xvGb7RtSmASgHFjLF7w/runs/0/artifacts/public/logs/live_backing.log

[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-PASS | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10rem - computed 
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vw - computed - assert_equals: expected "40px" but got "98px"
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vh - computed - assert_equals: expected "56px" but got "137.2px"
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vmin - computed - assert_equals: expected "40px" but got "98px"
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vmax - computed - assert_equals: expected "56px" but got "137.2px"
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → DUPLICATE
Reverting the changes. Looked a bit and it appears this failure on android started on this merge: 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2C7.0%2Cx86%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-android-em-7.0-x86%2Fopt-web-platform-tests-23%2Cw%28wpt23%29&revision=c5b713000513a2cdc1fdbc70aeb8f7d78bd687b2

Emilio, can it be from  Bug 775618 or from any of your bugs in that merge?
Status: RESOLVED → REOPENED
Flags: needinfo?(emilio)
Resolution: DUPLICATE → ---
I'm moderately sure there's a similar test that started changing with bug 1509552, wouldn't surprise me if this was also related since the failures are related to viewport units.
Blocks: 1509552
Flags: needinfo?(emilio) → needinfo?(botond)
(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)
> I'm moderately sure there's a similar test that started changing with bug
> 1509552

Perhaps you're thinking of bug 1511624.
(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)
> I'm moderately sure there's a similar test that started changing with bug
> 1509552, wouldn't surprise me if this was also related since the failures
> are related to viewport units.

What's strange is that a local backout of bug 1509552 did not make this test start passing.

I've been trying to run the test locally to investigate further, but haven't had any luck so far; current issue I'm running into is tracked in bug 1513078.
Depends on: 1513078
(In reply to Botond Ballo [:botond] from comment #7)
> I've been trying to run the test locally to investigate further, but haven't
> had any luck so far; current issue I'm running into is tracked in bug
> 1513078.

And now bug 1513695.
Depends on: 1513695
No longer depends on: 1513078
Ok, I was able to run the test in the Android emulator and reproduce the failure. Will look into it.
No longer depends on: 1513695
The failure is almost certainly related to the fact that the test harness uses window.inner{Width,Height} to emulate computation of viewport-relative units [1]. This is problematic in Firefox, because viewport-relative units are interepreted relative to the layout viewport, while in Firefox window.inner{Width,Height} report visual viewport dimensions (which is a known compat issue, for which I (coincidentally) just filed bug 1513703 yesterday [2]).

However, I don't quite understand yet why bug 1509552 made a difference, as the compat issue is pre-existing.

[1] https://searchfox.org/mozilla-central/rev/49e78df13e7a505827a3a86daae9efdf827133c6/testing/web-platform/tests/css/css-shapes/shape-outside/values/support/parsing-utils.js#359
(In reply to Botond Ballo [:botond] from comment #10)
> However, I don't quite understand yet why bug 1509552 made a difference, as
> the compat issue is pre-existing.

So, what seems to have happened is:

  - The tests have always been failing, due to the described
    known compat issue, and were marked as such [1].

  - Bug 1509552 introduced a temporary regression, which, as
    a side effect, caused the tests to pass. As a result, an
    UNEXPECTED-PASS bug was filed (bug 1511324), and the
    failure annotations were removed.

  - Bug 1511375 fixed the temporary regression, causing the
    tests to fail again, thereby triggering this bug.

So, I believe we just need to re-add the failure annotations, perhaps with a comment pointing to the compat bug that's on file for the underlying issue (bug 1513703).
Flags: needinfo?(botond)
(In reply to Botond Ballo [:botond] from comment #11)
>   - The tests have always been failing, due to the described
>     known compat issue, and were marked as such [1].

Forgot the link:

[1] https://hg.mozilla.org/mozilla-central/file/084b61a81dc2/testing/web-platform/meta/css/css-shapes/shape-outside/values/shape-margin-001.html.ini#l21
(In reply to Botond Ballo [:botond] from comment #11)
> So, I believe we just need to re-add the failure annotations, perhaps with a
> comment pointing to the compat bug that's on file for the underlying issue
> (bug 1513703).

Actually, I filed a more specific bug for window.inner{Width,Height} (bug 1514429), it's better to link to that.
(In reply to Botond Ballo [:botond] from comment #11)
> So, I believe we just need to re-add the failure annotations, perhaps with a
> comment pointing to the compat bug that's on file for the underlying issue
> (bug 1513703).

It looks like bug 1511399 already re-added the annotations.
(In reply to Botond Ballo [:botond] from comment #15)
> (In reply to Botond Ballo [:botond] from comment #11)
> > So, I believe we just need to re-add the failure annotations, perhaps with a
> > comment pointing to the compat bug that's on file for the underlying issue
> > (bug 1513703).
> 
> It looks like bug 1511399 already re-added the annotations.

It would still be nice to add a comment pointing to bug 1513703, but it's not entirely clear to me whether the test expectation .ini files support comments.
Pushed by bballo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fe8375a4e345
Annotate some shape-margin-001.html test failures with the bug that will fix them. r=emilio
https://hg.mozilla.org/mozilla-central/rev/fe8375a4e345
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Assignee: nobody → botond
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: