Closed Bug 1525734 Opened Last year Closed 9 months ago

Review tier 3 tasks

Categories

(Testing :: General, enhancement, P2)

Version 3
enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Tier 3 tasks are pretty low-value: They take time and money to run, but does anyone use or notice their results? Let's check with long-running tier 3 task owners to see if any of these can be turned off.

Depends on: 1525738

Tier 3 provides value for performance tests, allowing perf regressions to be monitored even when the test is unreliable; consider https://bugzilla.mozilla.org/show_bug.cgi?id=1531441#c4, for instance.

Depends on: 1538560
Depends on: 1539250
Depends on: 1539990
Depends on: 1540241

Talos tests on -ccov builds were made tier 3 in bug 1423214. Some of those tests are perma-fail today; some are perma-green:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=3&searchStr=talos%2Cccov

:marco, :jmaher -- Is that okay? Do we still get value from all of those tasks? Do you want to keep running them at tier 3?

Flags: needinfo?(mcastelluccio)
Flags: needinfo?(jmaher)

I see that we still upload coverage data even on the failed jobs. I am not sure what comes from that. Overall, I am not sure if we are gaining useful or relevant coverage data from the orange jobs since we are not completing the test as we would normally.

Flags: needinfo?(jmaher)

I don't know if somebody is actually using the Talos coverage data. I can see it being useful to know what we are testing for performance and what we are not, but I don't know if somebody is actually using it.

Flags: needinfo?(mcastelluccio)
Flags: needinfo?(klahnakoski)
Flags: needinfo?(gmierz2)

I haven't seen the Talos CCOV tasks used for anything in performance-testing coverage yet and I haven't heard of any plans to do so either (:ekyle may know more about this though). Based on that, to me, it makes sense to disable those on mozilla-central and leave them as an option for try pushes if we want.

Flags: needinfo?(gmierz2)

My understanding was that the Talos coverage was running to prove it can be done. We certainly had plans to use it to help with scheduling the right job based on changeset, but that seems stalled. I have no problems turning this off in central. My only concern is it would rot while we are not looking.

Flags: needinfo?(klahnakoski)
Depends on: 1543826

Some tier 3 tasks were recently added intentionally. I assume these are running as tier 3 temporarily. I won't make any changes to these at this time:

Depends on: 1544113
Depends on: 1543993

All done here. We stopped running some tasks and moved others to tier 1 or 2. Some tasks continue as tier 3, but I'm happy we've verified they are running purposefully.

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.