B2G KK Emulator opt showing up as Tier 1 when it should not

RESOLVED FIXED

Status

Tree Management
Treeherder
P2
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: Tomcat, Unassigned)

Tracking

Details

(URL)

(Reporter)

Description

2 years ago
for some reason in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=622952a12884 B2G KK Emulator opt was showing up for 2 pushes as tier 1 job, when it should be tier 3

no idea why this happened
(Reporter)

Comment 1

2 years ago
cameron, do you know why this happened here ?
Flags: needinfo?(cdawson)
This was a mistake in the Tier-3 profile.  We fixed it and I think things are back to normal now.  Please reopen if not.  :)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Flags: needinfo?(cdawson)
Resolution: --- → FIXED
(Reporter)

Comment 3

2 years ago
and same problem back in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=dc37c0e8f1351a5cd08960a53cfcc9cf753d26dc , Cameron can you take a look again ?
Status: RESOLVED → REOPENED
Flags: needinfo?(cdawson)
Resolution: FIXED → ---
this is really odd.  Yeah, several pushes have them as Tier-1.  But then the ones right after the push you mentioned are set correctly:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=b2g%20kk%20emulator%20opt&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=88df6d5a1f21bd9c8298f3503620a9647ef3d857&tochange=364b20e86805a62113e5262a0292a6a15dbfff1b

I went ahead and re-saved the Tier-3 exclusion profile anyway.  But it seemed to have fixed itself before I even got to it.  

Would you please ping me if it happens again?  Maybe I can find a pattern of some kind.
Flags: needinfo?(cdawson)
It looks like the reference_data_signature of those jobs changed.  This is what we anchor on to decide whether or not to apply Tier-3 (or whatever Tier) to a job.

Old signature: "77f047f34b6f2613a0066b59c858d4e00bb00da6"
New signature: "1c9ee226d7e7e6eaa018783d52bdf2eac85f99d9"

The signature hash is generated on several string value properties of a job as it comes in.  These ones changed from (during ingestion) job_group_name "Reftest" to "Mulet Reftest" and job_group_symbol from "tc-R" to just "R".  

We pick up the new values in the UI even for older jobs, but the signatures are different as you can see here:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=B2G%20KK%20Emulator%20opt%20Mulet%20Reftest%20%5BTC%5D%20Reftest%20R(R1)&filter-tier=1&filter-tier=2&filter-tier=3&tochange=bccb11375f2af838cda714d42fd8cef78f5c7bf1

Anyway, the long and short is that to update the Tier-3 signatures when something changes, you just need to have a Sheriff go to the "Sheriffing" panel, edit the "Tier-3" profile and click save.  That will re-evaluate the list of jobs that should be Tier-3.

This system is ending up being fairly brittle, as this situation shows.  I think this is ripe for a re-think on how to make this more robust.  Let's keep an eye on this situation and perhaps address this or next quarter as needed.  It'll be a fair bit of work.

Updated

2 years ago
Severity: normal → major
Priority: -- → P2
Bumping the priority on this bug.  It may go P1 if it becomes more frequent and really slows the sheriffs down.
(Reporter)

Comment 8

2 years ago
(In reply to Cameron Dawson [:camd] from comment #7)
> Bumping the priority on this bug.  It may go P1 if it becomes more frequent
> and really slows the sheriffs down.

it just came back on m-c this time btw
Cameron, what are the next steps here? It sounds like this one off issue was fixed, but the underlying cause not.

Should we close this out as fixed (since it was about the one-off) and file a new bug with clearer comment 0 about the long term fix? :-)
Flags: needinfo?(cdawson)
Ed-- Yeah, that makes sense.  I'll create a new bug tomorrow.
Tomcat: was this then remedied by re-saving the Tier-3 exclusion profile?

I created bug 1277993 to encompass re-thinking this feature.
Flags: needinfo?(cdawson) → needinfo?(cbook)
(Reporter)

Comment 12

2 years ago
yeah i think that saved us then again
Flags: needinfo?(cbook)
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.