1542825 - Inconsistent results on specific web platform tests

Status: NEW

(Testing :: web-platform-tests, defect, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

I'm testing css/css-shapes/shape-outside/shape-image/gradients/shape-outside-linear-gradient-003.html (and the rest of the tests listed in bug 1525705) on Windows with and without WebRender.

Currently, in automation, the tests are marked as failing with WebRender enabled, but passing with WebRender disabled. However, I get the opposite result when I run locally, like so:

WebRender enabled:
MOZ_WEBRENDER=1 ./mach wpt testing/web-platform/tests/css/css-shapes/shape-outside/shape-image/gradients/shape-outside-linear-gradient-003.html

WebRender disabled:
MOZ_WEBRENDER=0 ./mach wpt testing/web-platform/tests/css/css-shapes/shape-outside/shape-image/gradients/shape-outside-linear-gradient-003.html

Additionally, if I just load the test file in the browser directly with WR enabled, and modify the test file to pick up an Ahem.ttf file properly, I see the correct expected results with WebRender enabled.

It's not clear to me why automation is producing the opposite result from my local runs. I also can't seem to find a way to debug this locally because after I run the wpt locally, I can't use devtools on the window that shows the reftest so I can't inspect the layout properties of the various boxes and see why it might be wrong.

Comment 1

[James Graham [:jgraham]]

OK, so there are various issues here

Currently, in automation, the tests are marked as failing with WebRender enabled, but passing with WebRender disabled. However, I get the opposite result when I run locally

I don't have Windows available to try this out (it seems like everything passes as expected on Linux). I presume you are sure that you aren't seeing an expected-fail that you take to be a pass? (using --log-tbpl - might make this more explict).

Additionally, if I just load the test file in the browser directly with WR enabled, and modify the test file to pick up an Ahem.ttf file properly, I see the correct expected results with WebRender enabled.

You don't need to mosdify the test file; if you cd testing/web-platform/tests && ./wpt serve that will start the server environment so you can load the tests directly.

I also can't seem to find a way to debug this locally because after I run the wpt locally,

Is it possible to get anywhere by dumping the layout tree?

I can't use devtools on the window that shows the reftest so I can't inspect the layout properties of the various boxes and see why it might be wrong.

Hmm, I thought there was some way to do this with remote debugging, but I can't make it work now. Does it work with the reftest harness?

Anyway I think you want a try push with all your failing tests that passes --reftest-screenshot=fail (or =always) to the harness. That way you can at least see the failures in the reftest analyzer.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

(In reply to James Graham [:jgraham] from comment #1)

Currently, in automation, the tests are marked as failing with WebRender enabled, but passing with WebRender disabled. However, I get the opposite result when I run locally

I don't have Windows available to try this out (it seems like everything passes as expected on Linux).

Yes, on Linux and mac I get the correct behaviour with and without WR

I presume you are sure that you aren't seeing an expected-fail that you take to be a pass? (using --log-tbpl - might make this more explict).

No, I get an UNEXPECTED-PASS with WR enabled (because the expectations file says it should be failing, as it does in automation). And I get a TEST-FAIL with WR disabled.

Additionally, if I just load the test file in the browser directly with WR enabled, and modify the test file to pick up an Ahem.ttf file properly, I see the correct expected results with WebRender enabled.

You don't need to mosdify the test file; if you cd testing/web-platform/tests && ./wpt serve that will start the server environment so you can load the tests directly.

Ah, good to know, thanks. I was looking for a ./mach wpt-serve since I recalled that there was some way to do this, but I didn't realize it was a separate script.

I also can't seem to find a way to debug this locally because after I run the wpt locally,

Is it possible to get anywhere by dumping the layout tree?

Yeah. If necessary I can debug in Gecko by setting various logging prefs and printlns. It's just that being able to use the devtools would help narrow down the problem and reduce the scope of what I would need to do in Gecko.

I can't use devtools on the window that shows the reftest so I can't inspect the layout properties of the various boxes and see why it might be wrong.

Hmm, I thought there was some way to do this with remote debugging, but I can't make it work now. Does it work with the reftest harness?

In the (non-WPT) reftest harness it usually doesn't keep the test page open, so I don't think I've ever used devtools there. But I thought it might work with WPT since it seems to leave the window open with the rendering.

Anyway I think you want a try push with all your failing tests that passes --reftest-screenshot=fail (or =always) to the harness. That way you can at least see the failures in the reftest analyzer.

I did do a try push (see bug 1525705 comment 2 for the reftest analyzer link). And the failures that are observed in the try push with WR enabled are what I'm seeing locally with WR disabled.

At this point though I'm more concerned with the disparity between what happens on try and what I get locally, because I suspect fixing that might just make the test pass consistently everywhere like it does on Mac or Linux. I guess there's not much to do but to debug the failure locally via Gecko instrumentation and see why it's failing, and that might then shed light on why I get different results.

Comment 3

[BugBot [:suhaib / :marco]]

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[James Graham [:jgraham]]

I'm not sure what the remaining actionable issues are here; did you make any progress on understanding the disparity?

Comment 5

[Kartikaya Gupta (email:[e-mail])]

The action item here is to investigate why I get different results on my local setup than on try. But I don't have cycles at the moment to do that investigation.