1511915 - Perma Fail Tier 2 Android /css/css-shapes/shape-outside/values/shape-margin-001.html | XXXX - computed - assert_equals: expected "XXXpx" but got "XXXpx"

Status: RESOLVED FIXED

(Core :: CSS Parsing and Computation, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=215346752&repo=mozilla-central

https://queue.taskcluster.net/v1/task/bb5xvGb7RtSmASgHFjLF7w/runs/0/artifacts/public/logs/live_backing.log

[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-PASS | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10rem - computed 
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vw - computed - assert_equals: expected "40px" but got "98px"
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vh - computed - assert_equals: expected "56px" but got "137.2px"
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.549Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.550Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vmin - computed - assert_equals: expected "40px" but got "98px"
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9
[task 2018-12-03T23:26:07.554Z] 23:26:07     INFO - setupFonts/<@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:462:13
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests/</<@http://web-platform.test:8000/resources/testharness.js:769:30
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1568:20
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - test@http://web-platform.test:8000/resources/testharness.js:544:21
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests/<@http://web-platform.test:8000/resources/testharness.js:767:21
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - forEach@http://web-platform.test:8000/resources/testharness.js:3099:17
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - generate_tests@http://web-platform.test:8000/resources/testharness.js:764:9
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - @http://web-platform.test:8000/css/css-shapes/shape-outside/values/shape-margin-001.html:30:9
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - 
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - TEST-UNEXPECTED-FAIL | /css/css-shapes/shape-outside/values/shape-margin-001.html | 10vmax - computed - assert_equals: expected "56px" but got "137.2px"
[task 2018-12-03T23:26:07.555Z] 23:26:07     INFO - testShapeMarginComputedStyle@http://web-platform.test:8000/css/css-shapes/shape-outside/values/support/parsing-utils.js:60:9

Comment 1

[Intermittent Failures Robot]

19 failures in 206 pushes (0.092 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 19

Platform breakdown:
* android-em-7-0-x86: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1511915&startday=2018-12-03&endday=2018-12-03&tree=all

Comment 3

[Andreea Pavel [:apavel]]

Reverting the changes. Looked a bit and it appears this failure on android started on this merge: 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2C7.0%2Cx86%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-android-em-7.0-x86%2Fopt-web-platform-tests-23%2Cw%28wpt23%29&revision=c5b713000513a2cdc1fdbc70aeb8f7d78bd687b2

Emilio, can it be from  Bug 775618 or from any of your bugs in that merge?

Comment 4

[Emilio Cobos Álvarez (:emilio)]

I'm moderately sure there's a similar test that started changing with bug 1509552, wouldn't surprise me if this was also related since the failures are related to viewport units.

Comment 5

[Intermittent Failures Robot]

19 failures in 1862 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 19

Platform breakdown:
* android-em-7-0-x86: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1511915&startday=2018-12-03&endday=2018-12-09&tree=all

Comment 6

[Botond Ballo [:botond]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)
> I'm moderately sure there's a similar test that started changing with bug
> 1509552

Perhaps you're thinking of bug 1511624.

Comment 7

[Botond Ballo [:botond]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)
> I'm moderately sure there's a similar test that started changing with bug
> 1509552, wouldn't surprise me if this was also related since the failures
> are related to viewport units.

What's strange is that a local backout of bug 1509552 did not make this test start passing.

I've been trying to run the test locally to investigate further, but haven't had any luck so far; current issue I'm running into is tracked in bug 1513078.

Comment 8

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #7)
> I've been trying to run the test locally to investigate further, but haven't
> had any luck so far; current issue I'm running into is tracked in bug
> 1513078.

And now bug 1513695.

Comment 9

[Botond Ballo [:botond]]

Ok, I was able to run the test in the Android emulator and reproduce the failure. Will look into it.

Comment 10

[Botond Ballo [:botond]]

The failure is almost certainly related to the fact that the test harness uses window.inner{Width,Height} to emulate computation of viewport-relative units [1]. This is problematic in Firefox, because viewport-relative units are interepreted relative to the layout viewport, while in Firefox window.inner{Width,Height} report visual viewport dimensions (which is a known compat issue, for which I (coincidentally) just filed bug 1513703 yesterday [2]).

However, I don't quite understand yet why bug 1509552 made a difference, as the compat issue is pre-existing.

[1] https://searchfox.org/mozilla-central/rev/49e78df13e7a505827a3a86daae9efdf827133c6/testing/web-platform/tests/css/css-shapes/shape-outside/values/support/parsing-utils.js#359

Comment 11

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #10)
> However, I don't quite understand yet why bug 1509552 made a difference, as
> the compat issue is pre-existing.

So, what seems to have happened is:

  - The tests have always been failing, due to the described
    known compat issue, and were marked as such [1].

  - Bug 1509552 introduced a temporary regression, which, as
    a side effect, caused the tests to pass. As a result, an
    UNEXPECTED-PASS bug was filed (bug 1511324), and the
    failure annotations were removed.

  - Bug 1511375 fixed the temporary regression, causing the
    tests to fail again, thereby triggering this bug.

So, I believe we just need to re-add the failure annotations, perhaps with a comment pointing to the compat bug that's on file for the underlying issue (bug 1513703).

Comment 13

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #11)
>   - The tests have always been failing, due to the described
>     known compat issue, and were marked as such [1].

Forgot the link:

[1] https://hg.mozilla.org/mozilla-central/file/084b61a81dc2/testing/web-platform/meta/css/css-shapes/shape-outside/values/shape-margin-001.html.ini#l21

Comment 14

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #11)
> So, I believe we just need to re-add the failure annotations, perhaps with a
> comment pointing to the compat bug that's on file for the underlying issue
> (bug 1513703).

Actually, I filed a more specific bug for window.inner{Width,Height} (bug 1514429), it's better to link to that.

Comment 15

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #11)
> So, I believe we just need to re-add the failure annotations, perhaps with a
> comment pointing to the compat bug that's on file for the underlying issue
> (bug 1513703).

It looks like bug 1511399 already re-added the annotations.

Comment 16

[Botond Ballo [:botond]]

(In reply to Botond Ballo [:botond] from comment #15)
> (In reply to Botond Ballo [:botond] from comment #11)
> > So, I believe we just need to re-add the failure annotations, perhaps with a
> > comment pointing to the compat bug that's on file for the underlying issue
> > (bug 1513703).
> 
> It looks like bug 1511399 already re-added the annotations.

It would still be nice to add a comment pointing to bug 1513703, but it's not entirely clear to me whether the test expectation .ini files support comments.

Comment 17

[Emilio Cobos Álvarez (:emilio)]

You can add comments like: https://searchfox.org/mozilla-central/rev/47edbd91c43db6229cf32d1fc4bae9b325b9e2d0/testing/web-platform/meta/compat/webkit-appearance.html.ini#6

Comment 18

[Botond Ballo [:botond]]

Attachment: Bug 1511915 - Annotate some shape-margin-001.html test failures with the bug that will fix them. r=emilio

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fe8375a4e345
Annotate some shape-margin-001.html test failures with the bug that will fix them. r=emilio

Comment 20

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/fe8375a4e345