1416696 - Enable AWFY for performance sheriffing

Status: RESOLVED WONTFIX

(Testing Graveyard :: AWFY, enhancement)

Description

[Ionuț Goldan [:igoldan]]

We must address following steps to properly sheriff AWFY alerts:

1. Enable job retriggering
2. Triage alerts from last 2-3 months
3. Identify unstable tests (on specific platforms)
5. Gather docs for all our stable tests & publish them
6. Disable unstable tests

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

#1 will be difficult. you can do it with AWFY, but right now it isn't from treeherder.  Also there is a lack of backfilling on AWFY and I don't think that exists currently.  Related to this topic is load on the machines- Most of these machines have much fewer available CPU cycles- so getting many extra runs in will be hard.

I would solve 3/5/6 in your list (there is a 4 missing) and then we can delete old alerts and start triaging from a given point in time.

Comment 2

[Ed Morley [:emorley]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1)
> #1 will be difficult. you can do it with AWFY, but right now it isn't from
> treeherder.  Also there is a lack of backfilling on AWFY and I don't think
> that exists currently.  Related to this topic is load on the machines- Most
> of these machines have much fewer available CPU cycles- so getting many
> extra runs in will be hard.

This kind of thing (feature parity with Taskcluster jobs) is only going to get harder for third parties I think. I've posted to the Taskcluster newsgroup on this topic:
https://groups.google.com/forum/#!topic/mozilla.tools.taskcluster/Stpw9bKq-h4

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:igoldan, can you update here with where we are at and what are the next steps?

Comment 4

[Ionuț Goldan [:igoldan]]

I didn't manage to look over this issue yet.

Comment 5

[Ionuț Goldan [:igoldan]]

I'm working on finalizing bug 1400380.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I think we need to revisit this in Q2- we might reduce the number of benchmarks (and also add some different ones)- likewise we need the ability to push to try and run locally- we either need to fix AWFY to make it modern, or create a tier-2 sheriffing model.

Comment 7

[Bob Clary [:bc] (inactive)]

AWFY is dead. Long live AWFY.

Comment 8

[Ionuț Goldan [:igoldan]]

Dave, could we remove AWFY entirely from Perfherder?

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan] from comment #8)

Dave, could we remove AWFY entirely from Perfherder?

Yes, please do.