stylo: turn reftest-stylo jobs on by default but make them tier 3 (and so hidden by default)

RESOLVED FIXED

Status

()

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: heycam, Assigned: heycam)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
reftest-stylo jobs aren't running by default since we moved from incubator/stylo, but there are still a couple of leaks we need to sort out before they're all green.  Let's get them running but make them tier 3 for now.
(Assignee)

Comment 1

2 years ago
Created attachment 8837111 [details] [diff] [review]
patch
Assignee: nobody → cam
Status: NEW → ASSIGNED
Attachment #8837111 - Flags: review?(dustin)
Comment on attachment 8837111 [details] [diff] [review]
patch

Huh, it looks to me like this patch would run reftest-stylo on all platforms. Don't we still want to limit it to linux64-stylo?

I would expect this patch to remove some annotation limiting the jobs to the "stylo" tree on treeherder. But maybe I don't understand how all this works?
Flags: needinfo?(kmoir)
Right, this patch would run reftest-stylo on all platforms.  We don't want this because 1) it's expensive 2) we don't have stylo builds running on integration branches other than linux64 on m-c
Flags: needinfo?(kmoir)
(In reply to Kim Moir [:kmoir] from comment #4)
> Right, this patch would run reftest-stylo on all platforms.  We don't want
> this because 1) it's expensive 2) we don't have stylo builds running on
> integration branches other than linux64 on m-c

Yeah, there's zero value in running these tests on non-stylo builds.

That said, I'm a bit confused as to why these jobs aren't showing up on central given that the existing configuration looks right to me. Anything obvious?
Flags: needinfo?(kmoir)
Comment on attachment 8837111 [details] [diff] [review]
patch

Review of attachment 8837111 [details] [diff] [review]:
-----------------------------------------------------------------

reftest-stylo is only included in the stylo-tests test platform:
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-sets.yml#68
which only runs on the stylo platforms:
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#62
so this change does not add any additional platforms.

However, it would include these tests on any projects that build stylo, where previously the tests would have been omitted by run-on-projects: [].  I suspect you only want to run these on a few projects (maybe just try? or stylo and stylo-try?), rather than the default ['all'].
Attachment #8837111 - Flags: review?(dustin)
We want to run these jobs for the linux64-stylo platforms on autoland, inbound, and central, just like we seem to be doing for mochitest-style. I still don't quite understand why that isn't happening right now.
As mentioned in other quantum bugs, budget was not allocated to run additional tests for quantum.  As a result we have asked people to limit them to running on m-c to reduce the budget impact.  Running these builds on every commit is expensive on high volume branches like m-i, autoland, try. Running these tests on every commit even more so. Could we limit builds and tests tor run on m-c to reduce the budget impact?
Flags: needinfo?(kmoir)
Depends on: 1339604
(In reply to Kim Moir [:kmoir] from comment #8)
> As mentioned in other quantum bugs, budget was not allocated to run
> additional tests for quantum.  As a result we have asked people to limit
> them to running on m-c to reduce the budget impact.  Running these builds on
> every commit is expensive on high volume branches like m-i, autoland, try.
> Running these tests on every commit even more so. Could we limit builds and
> tests tor run on m-c to reduce the budget impact?

We really can't keep our jobs green if we do that.

These jobs are cheap - a small fraction of the stuff we run on normal linux64. Assuming our monthly AWS bill is still ~five figures, the cost of these additional jobs is dwarfed by the headcount cost we're spending on improving quantum productivity, let alone the headcount cost of the people whose actual productivity is at stake.

I know the budgeting issue isn't your fault, so let me know if you get pushback over this and I will escalate.
(Let's move any further discussion of the budgeting issue over to bug 1339604)
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.