1520946 - Investigate why the WPT copy of flexbox-justify-content-horiz-001b.xhtml is failing on most platforms

Status: RESOLVED FIXED

(Core :: Layout: Flexbox, defect, P3)

Description

[Daniel Holbert [:dholbert]]

For some reason, the WPT copy of flexbox-justify-content-horiz-001b.xhtml is annotated as failing:
https://searchfox.org/mozilla-central/rev/dac799c9f4e9f5f05c1071cba94f2522aa31f7eb/testing/web-platform/meta/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-justify-content-horiz-001b.xhtml.ini

However, the version in our layout/reftests directory passes just fine (no fails annotations):
https://searchfox.org/mozilla-central/rev/dac799c9f4e9f5f05c1071cba94f2522aa31f7eb/layout/reftests/w3c-css/submitted/flexbox/reftest.list#163

And interestingly, the "failing" WPT copy just started passing on "Android 7.0 x86", as of this push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=e3cb5a5ef667973b60314b67e13184f5d9cedb18&searchStr=Android%2C7.0%2Cx86%2CWr5&selectedJob=222383281

My current theory is that the "failure" is due to scrollbars (since I do see a barely-necessary scrollbar when I run the WPT version of the test locally), and I suspect it started passing on android due to one of the changes on bug 1504659 (which impacted when scrollbars appear on Android).

If my theory's correct, we could perhaps make the test pass everywhere by shrinking its contents' height slightly so that no scrollbar is ever needed. I don't immediately know how to see screenshots of the failure (./mach test [path-to-test] doesn't give me any, unfortunately), so that's why my diagnosis is just a theory at this point.

Comment 1

[Daniel Holbert [:dholbert]]

Ah, I think I know what's wrong here -- it looks like flexbox-justify-content-horiz-001b.xhtml and its friends all have "margin-bottom: 2px;" on their flex containers, but in -001b, the margin-bottom at the very end isn't creating scrollable overflow (perhaps because the flex container is floated?)

So there's a very slightly different page-height between the test & reference, & a different size scrollbar as a result.

Comment 2

[Daniel Holbert [:dholbert]]

Attachment: Bug 1520946: Shrink some content in various w3c-submitted flexbox tests so that they won't trigger scrollbars in WPT harness. r?TYLin

All of these tests have an .ini file indicating a failure for at least one of
them right now, in the copy that lives in the WPT harness, because

• they trigger a scrollbar in the WPT harness
  ...and:
• the scrollbar is a slightly different size in the reference case vs.
  in one of the testcases, e.g. due to a tiny margin being honored (or not)
  on the final piece of content.

This patch shrinks the content a bit in these test files so that they all fit
in a 400x400 viewport and won't trigger a scrollbar. I expect this should make
them pass in the WPT harness when they make the roundtrip through our
synchronization process.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1b73f4cd7208
Shrink some content in various w3c-submitted flexbox tests so that they won't trigger scrollbars in WPT harness. r=TYLin

Comment 5

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/1b73f4cd7208

Comment 6

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

WPT PR in https://github.com/web-platform-tests/wpt/pull/14965

Comment 7

[Intermittent Failures Robot]

1 failures in 2378 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1520946&startday=2019-01-14&endday=2019-01-20&tree=all