1570906 - Performance alert created from a [tier 3] job

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

This alert was created from a [tier 3] job, and wasn't supposed to. IIRC, we keep unstable jobs on [tier 3] until they settle and then move them to upper tiers.
Eric, could you look into and eventually take the necessary actions, please?
Thaanks!

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a task, but please change it back in case of error.

Comment 2

[Bob Clary [:bc] (inactive)]

igoldan: This sounds like a treeherder/perfherder issue with alerting a tier 3 job rather than an issue with AWSY. I don't see anything in treeherder's repo under perf, perfalert which mentions tier.

Comment 3

[Eric Rahm [:erahm]]

I also don't have a problem with this alerting, it's tier3 because mitmproxy is flaky, not because the data is bad.

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Bob Clary [:bc:] from comment #2)

igoldan: This sounds like a treeherder/perfherder issue with alerting a tier 3 job rather than an issue with AWSY. I don't see anything in treeherder's repo under perf, perfalert which mentions tier.

Indeed, Perfherder doesn't check for the test's tier, when it should. I believe we need to address this.
Dave, should I file a separate bug to work on this?

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Let's use this bug and just update the product/component. Before we work on this I'd like to publish an update of our performance regression policy, which will stipulate which tiers we monitor for regressions.

Comment 6

[Bob Clary [:bc] (inactive)]

Over to perfherder.

Comment 7

[Ionuț Goldan [:igoldan]]

Have we decided about the right course of action here?
Should we stop alerting on tier 3 jobs?

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan] from comment #7)

Have we decided about the right course of action here?
Should we stop alerting on tier 3 jobs?

Yes, we should stop alerting on tier 3 jobs. I'm going to block this on updating the performance regression policy to explicitly mention that we do not sheriff jobs running as tier 3.

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/7037

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/7038