1525734 - Review tier 3 tasks

Status: RESOLVED FIXED

(Testing :: General, enhancement, P2)

Description

[Geoff Brown [:gbrown]]

Tier 3 tasks are pretty low-value: They take time and money to run, but does anyone use or notice their results? Let's check with long-running tier 3 task owners to see if any of these can be turned off.

Comment 1

[Geoff Brown [:gbrown]]

Tier 3 provides value for performance tests, allowing perf regressions to be monitored even when the test is unreliable; consider https://bugzilla.mozilla.org/show_bug.cgi?id=1531441#c4, for instance.

Comment 2

[Geoff Brown [:gbrown]]

Talos tests on -ccov builds were made tier 3 in bug 1423214. Some of those tests are perma-fail today; some are perma-green:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=3&searchStr=talos%2Cccov

:marco, :jmaher -- Is that okay? Do we still get value from all of those tasks? Do you want to keep running them at tier 3?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I see that we still upload coverage data even on the failed jobs. I am not sure what comes from that. Overall, I am not sure if we are gaining useful or relevant coverage data from the orange jobs since we are not completing the test as we would normally.

Comment 4

[Marco Castelluccio [:marco]]

I don't know if somebody is actually using the Talos coverage data. I can see it being useful to know what we are testing for performance and what we are not, but I don't know if somebody is actually using it.

Comment 5

[Greg Mierzwinski [:sparky]]

I haven't seen the Talos CCOV tasks used for anything in performance-testing coverage yet and I haven't heard of any plans to do so either (:ekyle may know more about this though). Based on that, to me, it makes sense to disable those on mozilla-central and leave them as an option for try pushes if we want.

Comment 6

[Kyle Lahnakoski [:ekyle]]

My understanding was that the Talos coverage was running to prove it can be done. We certainly had plans to use it to help with scheduling the right job based on changeset, but that seems stalled. I have no problems turning this off in central. My only concern is it would rot while we are not looking.

Comment 7

[Geoff Brown [:gbrown]]

Some tier 3 tasks were recently added intentionally. I assume these are running as tier 3 temporarily. I won't make any changes to these at this time:

• Windows Bg tasks (builds on gcp) - bug 1536555
• Raptor tp6-8-404 tasks - bug 1457274

Comment 8

[Geoff Brown [:gbrown]]

All done here. We stopped running some tasks and moved others to tier 1 or 2. Some tasks continue as tier 3, but I'm happy we've verified they are running purposefully.