1367636 - stylo: stylo-to-stylo reftest

Status: RESOLVED FIXED

(Core :: CSS Parsing and Computation, enhancement, P1)

Description

[Shing Lyu [:shinglyu]]

We've been running stylo-vs-gecko reftest on linux64-stylo for a while. Now it's time to run stylo-vs-stylo reftest in parallel.

This involves:
1. Add a new test job to the CI manifest
2. Create a separate flag to distinguish stylo-vs-gecko and stylo-vs-stylo
  * I enabled this in Bug 1361645, but got invalidated by Bug 1364863, so I got to rebase 1361645 and land
3. Mark the reftest.list expectations.

Comment 2

[Bobby Holley (:bholley)]

In terms of reftest manifest flag names, I think we should replace the current fails-if(stylo) with fails-if(styloVsGecko), and use fails-if(stylo) for stylo-vs-stylo mode.

Comment 3

[Shing Lyu [:shinglyu]]

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #2)
> In terms of reftest manifest flag names, I think we should replace the
> current fails-if(stylo) with fails-if(styloVsGecko), and use fails-if(stylo)
> for stylo-vs-stylo mode.

That's the plan for Bug 1361645, I'll land it ASAP.

Comment 4

[Shing Lyu [:shinglyu]]

I tried to enable stylo-to-stylo reftest, doesn't look too bad. https://treeherder.mozilla.org/#/jobs?repo=try&revision=64090709c1c6eede128b1b010c86b56271126abc

Comment 5

[Shing Lyu [:shinglyu]]

Attachment: Bug 1367636 - Make reftest "stylo" condition match only non-styloVsGecko

mode

Review commit: https://reviewboard.mozilla.org/r/143156/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/143156/

Comment 6

[Shing Lyu [:shinglyu]]

Attachment: Bug 1367636 - Enable stylo-vs-stylo reftest on CI

Review commit: https://reviewboard.mozilla.org/r/143158/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/143158/

Comment 7

[Shing Lyu [:shinglyu]]

Attachment: Bug 1367636 - Update stylo-vs-stylo reftest expectations

Review commit: https://reviewboard.mozilla.org/r/143160/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/143160/

Comment 8

[Shing Lyu [:shinglyu]]

Chris, 

Here are the patch for enabling stylo-vs-stylo reftest. I'll be on PTO next week so I'm not very confident to land it right now. If this is very urgent please find someone who know how to update reftest expectation to help land it, because we might need to update some reftest expectations while rebasing.

Comment 9

[Chris Peterson [:cpeterson]]

(In reply to Shing Lyu [:shinglyu] from comment #8)
> Here are the patch for enabling stylo-vs-stylo reftest. I'll be on PTO next
> week so I'm not very confident to land it right now. If this is very urgent
> please find someone who know how to update reftest expectation to help land
> it, because we might need to update some reftest expectations while rebasing.

Awesome! Thanks. I'll find someone to land your patches. Is your reftest expectations patch all green (for the moment)? Or will the person who lands this next week need to do fix more reftest expectations?

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/90a28b364ba8
Make reftest "stylo" condition match only non-styloVsGecko mode. r=heycam
https://hg.mozilla.org/integration/autoland/rev/6044cf528f8c
Enable stylo-vs-stylo reftest on CI. r=heycam
https://hg.mozilla.org/integration/autoland/rev/8db66ad9798b
Update stylo-vs-stylo reftest expectations. r=heycam

Comment 11

[Cameron McCormack (:heycam)]

I pushed this directly to autoland, after updating the test expectations, since I had to base the patches on autoland, but I couldn't push to mozreview on autoland due to some issue with being based on a merge commit.

One notable reftest.list change is that I disabled layout/reftests/text-overflow/block-padding.html, because it was passing in sequential mode but failing in parallel mode, and there's no sandbox flag to condition that failure on.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c5f45aaf4c8
Some more rem-unit test disabling. r=me

Comment 13

[Chris Peterson [:cpeterson]]

Thanks, Cameron! I see the stylo-vs-stylo (R#) and stylo-vs-gecko (Rs#) reftests are now running on Treeherder.

Comment 14

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/90a28b364ba8
https://hg.mozilla.org/mozilla-central/rev/6044cf528f8c
https://hg.mozilla.org/mozilla-central/rev/8db66ad9798b
https://hg.mozilla.org/mozilla-central/rev/9c5f45aaf4c8