disable BBB talos tasks for linux64-stylo builds

RESOLVED FIXED

Status

defect
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: jmaher, Assigned: wcosta)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
we appear to be trying to schedule talos tasks for linux64-stylo when we run on regular pushes.  Currently we only have talos ready for linux64 opt and pgo, not the stylo builds.

here are the buildernames:
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos dromaeojs
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos tp5o
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g1
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos dromaeojs-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos chromez-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g2
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g3-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos chromez
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos other-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g4-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g1-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g4
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g2-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos other
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos g3
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos svgr
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos tp5o-e10s
Ubuntu HW 12.04 x64 mozilla-inbound stylo talos svgr-e10s


we need to special case this, in fact, I would rather whitelist the build types we BBB for talos jobs :)
(Assignee)

Updated

2 years ago
Assignee: nobody → wcosta
Status: NEW → ASSIGNED
Comment hidden (mozreview-request)
(Reporter)

Comment 2

2 years ago
mozreview-review
Comment on attachment 8841962 [details]
Bug 1343095: Disable talos for linux-stylo builds.

https://reviewboard.mozilla.org/r/116020/#review117450

this looks good, lets fix this and then work with the stylo team to get this turned on properly.
Attachment #8841962 - Flags: review?(jmaher) → review+

Comment 3

2 years ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3ef4e3e72e1e
Disable talos for linux-stylo builds. r=jmaher
(Reporter)

Comment 4

2 years ago
:shinglyu, we are turning these off as we run talos jobs for liux64* from taskcluster but we proxy them to buildbot.  This means that buildbot needs to have a definition in place and without it we get errors and have a lot of cleanup to do.

Reading in bug 1328765, I see that there is still pending work to get talos working on stylo builds.  When we discussed the quantum project we were told that we wouldn't be duplicating tests, in this case we have a limited pool of hardware and the stylo talos tests will need to be scheduled with a bit more thought.
(Reporter)

Updated

2 years ago
Flags: needinfo?(slyu)
(In reply to Joel Maher ( :jmaher) from comment #4)
> :shinglyu, we are turning these off as we run talos jobs for liux64* from
> taskcluster but we proxy them to buildbot.  This means that buildbot needs
> to have a definition in place and without it we get errors and have a lot of
> cleanup to do.
> 
> Reading in bug 1328765, I see that there is still pending work to get talos
> working on stylo builds.  When we discussed the quantum project we were told
> that we wouldn't be duplicating tests, in this case we have a limited pool
> of hardware and the stylo talos tests will need to be scheduled with a bit
> more thought.

The agreement that came out of last week was that we would run talos tests on the linux64-stylo job on mozilla-central only (not inbound or autoland). The overhead of doing that should be pretty negligible. Is that what the patch in this bug does?
Flags: needinfo?(slyu) → needinfo?(jmaher)
(Reporter)

Comment 6

2 years ago
thanks for the information.  A few things here to consider:
1) this was scheduled on all branches, but no branches, not even m-c was working
2) we need to add buildbot configs for the new talos jobs (ideally mozilla-central and try)
3) then we can add back in taskcluster configs to schedule the talos jobs (we can test on try)
4) we should consider only testing e10s, we are planning to disable all non-e10s testing when Firefox 57 starts on nightly, mid June - do weigh in if there is a specific need for non-e10s talos data
5) we do not generate alerts for mozilla-central, therefore any performance sheriffing is not done on mozilla-central data, will you be actively monitoring this data, or will you be collecting some historical data
6) if only historical data, I would recommend running this once/day on mozilla-central until we are ready for more.  This can be done via a taskcluster cron.yml task- we do this for valgrind and for code coverage already.

do let me know what your thoughts are on these specific topics- I would be happy to help get these tests running in the short term!
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) from comment #6)
> thanks for the information.  A few things here to consider:
> 1) this was scheduled on all branches, but no branches, not even m-c was
> working

We're aiming to bring things up as fast as we can. There's no need to run them on all branches, but running them a few times per day on m-c will help us narrow down regressions if the tests break.

> 2) we need to add buildbot configs for the new talos jobs (ideally
> mozilla-central and try)
> 3) then we can add back in taskcluster configs to schedule the talos jobs
> (we can test on try)

I don't really know what this involves, but hopefully you or shing do. m-c and try are the right places for these jobs. Does this work make more sense for you to do, or for shing to do?

> 4) we should consider only testing e10s, we are planning to disable all
> non-e10s testing when Firefox 57 starts on nightly, mid June - do weigh in
> if there is a specific need for non-e10s talos data

e10s-only should be fine.

> 5) we do not generate alerts for mozilla-central, therefore any performance
> sheriffing is not done on mozilla-central data, will you be actively
> monitoring this data, or will you be collecting some historical data

Mostly historical for now, but also catching regressions that break the talos harness.

> 6) if only historical data, I would recommend running this once/day on
> mozilla-central until we are ready for more.  This can be done via a
> taskcluster cron.yml task- we do this for valgrind and for code coverage
> already.


I would be ok with that if somebody is willing to manage the details, and if the results actually show up somewhere in the normal treeherder view (so that it's easy to notice bustage). Given that this is a single platform, and given the single-digit daily push volume on m-c, I figured that running on m-c pushes only would have negligible cost, and I'd generally prefer to keep things simple.

> 
> do let me know what your thoughts are on these specific topics- I would be
> happy to help get these tests running in the short term!
(Reporter)

Updated

2 years ago
Depends on: 1343316
(Reporter)

Comment 8

2 years ago
ok, so lets run every commit on m-c, + ability for try; e10s only is good.

I filed bug 1343316 to handle the buildbot side of things, when that is done, we can use this bug to add the tests.
Keywords: leave-open
Sounds great - thanks!
Summary: disable BBB talos tasks for lnux64-stylo builds → disable BBB talos tasks for linux64-stylo builds
Blocks: 1338871
No longer blocks: 1338871
Depends on: 1338871
n/a since bug 1338871 has been closed.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.