1258720 - B2G KK Emulator opt showing up as Tier 1 when it should not

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Carsten Book [:Tomcat]]

for some reason in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=622952a12884 B2G KK Emulator opt was showing up for 2 pushes as tier 1 job, when it should be tier 3

no idea why this happened

Comment 1

[Carsten Book [:Tomcat]]

cameron, do you know why this happened here ?

Comment 2

[Cameron Dawson [:camd]]

This was a mistake in the Tier-3 profile.  We fixed it and I think things are back to normal now.  Please reopen if not.  :)

Comment 3

[Carsten Book [:Tomcat]]

and same problem back in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=dc37c0e8f1351a5cd08960a53cfcc9cf753d26dc , Cameron can you take a look again ?

Comment 4

[Cameron Dawson [:camd]]

this is really odd.  Yeah, several pushes have them as Tier-1.  But then the ones right after the push you mentioned are set correctly:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=b2g%20kk%20emulator%20opt&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=88df6d5a1f21bd9c8298f3503620a9647ef3d857&tochange=364b20e86805a62113e5262a0292a6a15dbfff1b

I went ahead and re-saved the Tier-3 exclusion profile anyway.  But it seemed to have fixed itself before I even got to it.  

Would you please ping me if it happens again?  Maybe I can find a pattern of some kind.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Saw this again today a few times.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&fromchange=09f42355aa061bada9ab92056215d9cdae86a220&group_state=expanded&filter-searchStr=kk
and
https://treeherder.mozilla.org/#/jobs?repo=fx-team&fromchange=09f42355aa061bada9ab92056215d9cdae86a220&group_state=expanded&selectedJob=8328614&filter-searchStr=kk

Comment 6

[Cameron Dawson [:camd]]

It looks like the reference_data_signature of those jobs changed.  This is what we anchor on to decide whether or not to apply Tier-3 (or whatever Tier) to a job.

Old signature: "77f047f34b6f2613a0066b59c858d4e00bb00da6"
New signature: "1c9ee226d7e7e6eaa018783d52bdf2eac85f99d9"

The signature hash is generated on several string value properties of a job as it comes in.  These ones changed from (during ingestion) job_group_name "Reftest" to "Mulet Reftest" and job_group_symbol from "tc-R" to just "R".  

We pick up the new values in the UI even for older jobs, but the signatures are different as you can see here:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=B2G%20KK%20Emulator%20opt%20Mulet%20Reftest%20%5BTC%5D%20Reftest%20R(R1)&filter-tier=1&filter-tier=2&filter-tier=3&tochange=bccb11375f2af838cda714d42fd8cef78f5c7bf1

Anyway, the long and short is that to update the Tier-3 signatures when something changes, you just need to have a Sheriff go to the "Sheriffing" panel, edit the "Tier-3" profile and click save.  That will re-evaluate the list of jobs that should be Tier-3.

This system is ending up being fairly brittle, as this situation shows.  I think this is ripe for a re-think on how to make this more robust.  Let's keep an eye on this situation and perhaps address this or next quarter as needed.  It'll be a fair bit of work.

Comment 7

[Cameron Dawson [:camd]]

Bumping the priority on this bug.  It may go P1 if it becomes more frequent and really slows the sheriffs down.

Comment 8

[Carsten Book [:Tomcat]]

(In reply to Cameron Dawson [:camd] from comment #7)
> Bumping the priority on this bug.  It may go P1 if it becomes more frequent
> and really slows the sheriffs down.

it just came back on m-c this time btw

Comment 9

[Ed Morley [:emorley]]

Cameron, what are the next steps here? It sounds like this one off issue was fixed, but the underlying cause not.

Should we close this out as fixed (since it was about the one-off) and file a new bug with clearer comment 0 about the long term fix? :-)

Comment 10

[Cameron Dawson [:camd]]

Ed-- Yeah, that makes sense.  I'll create a new bug tomorrow.

Comment 11

[Cameron Dawson [:camd]]

Tomcat: was this then remedied by re-saving the Tier-3 exclusion profile?

I created bug 1277993 to encompass re-thinking this feature.

Comment 12

[Carsten Book [:Tomcat]]

yeah i think that saved us then again