1339396 - stylo: turn reftest-stylo jobs on by default but make them tier 3 (and so hidden by default)

Status: RESOLVED FIXED

(Core :: CSS Parsing and Computation, defect)

Description

[Cameron McCormack (:heycam)]

reftest-stylo jobs aren't running by default since we moved from incubator/stylo, but there are still a couple of leaks we need to sort out before they're all green.  Let's get them running but make them tier 3 for now.

Comment 1

[Cameron McCormack (:heycam)]

Attachment: patch

Comment 2

[Cameron McCormack (:heycam)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=41615a0b31ac8b5b7217e59de7b776007235eb6a

Comment 3

[Bobby Holley (:bholley)]

Comment on attachment 8837111 [details] [diff] [review]
patch

Huh, it looks to me like this patch would run reftest-stylo on all platforms. Don't we still want to limit it to linux64-stylo?

I would expect this patch to remove some annotation limiting the jobs to the "stylo" tree on treeherder. But maybe I don't understand how all this works?

Comment 4

[Kim Moir [:kmoir] ET]

Right, this patch would run reftest-stylo on all platforms.  We don't want this because 1) it's expensive 2) we don't have stylo builds running on integration branches other than linux64 on m-c

Comment 5

[Bobby Holley (:bholley)]

(In reply to Kim Moir [:kmoir] from comment #4)
> Right, this patch would run reftest-stylo on all platforms.  We don't want
> this because 1) it's expensive 2) we don't have stylo builds running on
> integration branches other than linux64 on m-c

Yeah, there's zero value in running these tests on non-stylo builds.

That said, I'm a bit confused as to why these jobs aren't showing up on central given that the existing configuration looks right to me. Anything obvious?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8837111 [details] [diff] [review]
patch

Review of attachment 8837111 [details] [diff] [review]:
-----------------------------------------------------------------

reftest-stylo is only included in the stylo-tests test platform:
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#68
which only runs on the stylo platforms:
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#62
so this change does not add any additional platforms.

However, it would include these tests on any projects that build stylo, where previously the tests would have been omitted by run-on-projects: [].  I suspect you only want to run these on a few projects (maybe just try? or stylo and stylo-try?), rather than the default ['all'].

Comment 7

[Bobby Holley (:bholley)]

We want to run these jobs for the linux64-stylo platforms on autoland, inbound, and central, just like we seem to be doing for mochitest-style. I still don't quite understand why that isn't happening right now.

Comment 8

[Kim Moir [:kmoir] ET]

As mentioned in other quantum bugs, budget was not allocated to run additional tests for quantum.  As a result we have asked people to limit them to running on m-c to reduce the budget impact.  Running these builds on every commit is expensive on high volume branches like m-i, autoland, try. Running these tests on every commit even more so. Could we limit builds and tests tor run on m-c to reduce the budget impact?

Comment 9

[Bobby Holley (:bholley)]

(In reply to Kim Moir [:kmoir] from comment #8)
> As mentioned in other quantum bugs, budget was not allocated to run
> additional tests for quantum.  As a result we have asked people to limit
> them to running on m-c to reduce the budget impact.  Running these builds on
> every commit is expensive on high volume branches like m-i, autoland, try.
> Running these tests on every commit even more so. Could we limit builds and
> tests tor run on m-c to reduce the budget impact?

We really can't keep our jobs green if we do that.

These jobs are cheap - a small fraction of the stuff we run on normal linux64. Assuming our monthly AWS bill is still ~five figures, the cost of these additional jobs is dwarfed by the headcount cost we're spending on improving quantum productivity, let alone the headcount cost of the people whose actual productivity is at stake.

I know the budgeting issue isn't your fault, so let me know if you get pushback over this and I will escalate.

Comment 10

[Bobby Holley (:bholley)]

(Let's move any further discussion of the budgeting issue over to bug 1339604)