Open Bug 1381899 Opened 2 years ago Updated 2 years ago

Disable css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-min-width-auto-002c.html for instability

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(Not tracked)

People

(Reporter: jgraham, Unassigned)

References

Details

(Whiteboard: leave-open)

No description provided.
Hi! As the test author, I'm curious for a bit more detail here.  Could you elaborate on what instability you're talking about here?

It looks like this test is:
https://github.com/w3c/web-platform-tests/blob/master/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-min-width-auto-002c.html

...which I verified locally is the same as the original source material, already in mozilla-central:
https://dxr.mozilla.org/mozilla-central/source/layout/reftests/w3c-css/submitted/flexbox/flexbox-min-width-auto-002c.html

...which we run as a reftest and haven't annotated as random/fuzzy/failing/etc:
https://dxr.mozilla.org/mozilla-central/rev/5e73b9798464c3f7106f0161dc9a49b234f42f9c/layout/reftests/w3c-css/submitted/flexbox/reftest.list#158

So it comes as a bit of a surprise that we'd be unstable on this test, unless there's some sort of problem e.g. waiting for the image load/decode in the WPT harness (a problem which we apparently don't have under the normal reftest harness).
Flags: needinfo?(james)
So I did a run of just this directory [1]. Some metadata is wrong, but you can see that both 002a and 002c are unstable. With 002a I have a screenshot for use in the reftest analyzer, since it's not erroneously marked as expected FAIL. Since the logs will disappear after a while, it's below:

[task 2017-07-19T10:38:26.675477Z] 10:38:26     INFO - TEST-UNEXPECTED-FAIL | /css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-min-width-auto-002a.html | Testing http://web-platform.test:8000/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-min-width-auto-002a.html == http://web-platform.test:8000/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-min-width-auto-002-ref.html
[task 2017-07-19T10:38:26.677098Z] 10:38:26     INFO - REFTEST   IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAlgAAAJYCAYAAAC+ZpjcAAAK5klEQVR4nO3dwQnDMBQFQfWQguPOnRLyA0ssmxnwQTcdFx5Y6wQAILWuvgAAwNMILACAmMACAIgJLACAmMACAIgJLACAmMACAIh9DaxjHeexjn/cBQDgEQQWAEBsPBEKLQCAGYEFABAzEQIAxAQWAEDMRAgAEBNYAAAxEyEAQExgAQDETIQAADGBBQAQMxECAMQEFgBAzEQIABATWAAAMRMhAEBMYAEAxEyEAAAxgQUAEDMRAgDEBBYAQMxECAAQE1gAADETIQBATGABAMRMhAAAMYEFABAzEQIAxAQWAEDMRAgAEBNYAAAxEyEAQExgAQDExhMhAAAzAgsAIPbzRHj1GQBgdwILACA2ngjX633JBwBwNwILACA2ngivCiwTIQBwNwILACBmIgQAiAksAICYiRAAICawAABiJkIAgJjAAgCImQgBAGICCwAgZiIEAIgJLACAmIkQACAmsAAAYiZCAICYwAIAiJkIAQBiAgsAIGYiBACICSwAgJiJEAAgJrAAAGImQgCAmMACAIiZCAEAYgILACBmIgQAiAksAIDYeCLc5QwAsDuBBQAQG0+EAADMCCwAgJiJEAAgJrAAAGJ+0wAAEBNYAAAxT+UAAMQEFgBAzEQIABATWAAAMRMhAEBMYAEAxEyEAAAxgQUAEDMRAgDEBBYAQMxECAAQE1gAADETIQBATGABAMRMhAAAMYEFABAzEQIAxAQWAEDMRAgAEBNYAAAxEyEAQExgAQDETIQAADGBBQAQMxECAMQEFgBAzEQIABATWAAAsfFEuMsZAGB3AgsAIDaeCAEAmBFYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAAAxgQUAEBNYAACxDy5byaU48/OIAAAAAElFTkSuQmCC
[task 2017-07-19T10:38:26.678371Z] 10:38:26     INFO - REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAlgAAAJYCAYAAAC+ZpjcAAAK40lEQVR4nO3dsQnDQBBFwevBBVudyyV4gwdamxlQcJnCBx/uzg0AQOo8/QMAAP9GYAEAxAQWAEBMYAEAxAQWAEBMYAEAxAQWAEDsa2Bd57qvc605AwBsJ7AAAGLjifC83o98AAC/RmABAMTGE+FTgWUiBAB+jcACAIiZCAEAYgILACBmIgQAiAksAICYiRAAICawAABiJkIAgJjAAgCImQgBAGICCwAgZiIEAIgJLACAmIkQACAmsAAAYiZCAICYwAIAiJkIAQBiAgsAIGYiBACICSwAgJiJEAAgJrAAAGImQgCAmMACAIiZCAEAYgILACA2ngi3nAEAthNYAACx8UQIAMCMwAIAiJkIAQBiAgsAIOaaBgCAmMACAIh5KgcAICawAABiJkIAgJjAAgCImQgBAGICCwAgZiIEAIgJLACAmIkQACAmsAAAYiZCAICYwAIAiJkIAQBiAgsAIGYiBACICSwAgJiJEAAgJrAAAGImQgCAmMACAIiZCAEAYgILACBmIgQAiAksAICYiRAAICawAABiJkIAgJjAAgCIjSfCLWcAgO0EFgBAbDwRAgAwI7AAAGImQgCAmMACAIi5pgEAICawAABinsoBAIgJLACAmIkQACAmsAAAYiZCAICYwAIAiJkIAQBiAgsAIGYiBACICSwAgJiJEAAgJrAAAGImQgCAmMACAIiZCAEAYgILACBmIgQAiAksAICYiRAAICawAABiJkIAgJjAAgCImQgBAGICCwAgZiIEAIgJLACAmIkQACAmsAAAYuOJcMsZAGA7gQUAEBtPhAAAzAgsAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAICYwAIAiAksAIDYBxaM0aYngPhwAAAAAElFTkSuQmCC
[task 2017-07-19T10:38:26.678455Z] 10:38:26     INFO - TEST-INFO took 167ms

Seems like we aren't finishing laying out the top box in this case, which certainly seems like it must be a harness issue, but I don't entirely understand what's wrong because the implementation is supposed to be almost a copy of what's in the reftest harness. You can see the relevant code at [2] and [3]. We should be waiting for the load event and ensuring that layout is flushed using the same technique as the reftest harness. I can't obviously see what's wrong here, so pointers are very welcome.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=94b9ff39c8ce9d905895da2efd0099f45bb151a7&selectedJob=115583477
[2] http://searchfox.org/mozilla-central/source/testing/marionette/reftest.js#321
[3] http://searchfox.org/mozilla-central/source/testing/marionette/listener.js#1791
Flags: needinfo?(james)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a981b1a579c6
Disable wpt flexbox min-width test for instability, a=testonly
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cfccd9657b3d
fixup! Disable wpt flexbox min-width test for instability, a=testonly
Whiteboard: leave-open
(In reply to James Graham [:jgraham] from comment #2)
> I can't obviously see what's wrong here, so pointers are very welcome.

Nor can I :-/ Sorry.  I'm very curious, but I don't have cycles to dive in very much.
You need to log in before you can comment on or make changes to this bug.