1158892 - [raptor] Hide raptor tasks that post to TH production

Status: RESOLVED FIXED

(Tree Management Graveyard :: Visibility Requests, defect)

Description

[Robert Wood [:rwood]]

I am ready to turn on some raptor performance tests via taskcluster gaia, that will report to treeherder gaia production; however I would like to have the tasks hidden on TH please until we iron out any issues. Thanks!

There are three raptor tasks, each reporting to the same TH group, here's the info:

"treeherder": {
  "groupSymbol": "Rpt",
  "groupName": "Raptor Performance Tests",
  "symbol": "se1"
},

"treeherder": {
  "groupSymbol": "Rpt",
  "groupName": "Raptor Performance Tests",
  "symbol": "se2"
},

"treeherder": {
  "groupSymbol": "Rpt",
  "groupName": "Raptor Performance Tests",
  "symbol": "se"
},

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Jobs can't be hidden with Treeherder until they've run on a given tree.

Comment 2

[Robert Wood [:rwood]]

Ah, thanks Ryan. The Raptor tasks are now turned on and reporting to gaia TH production. If they could be hidden now please that'd be awesome, thanks!

https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=8ce7900c55bbc0dba4c2f687ebc7c15e4838c954

Comment 3

[Robert Wood [:rwood]]

Note: They're currently on TH gaia under 'Rpt' emulator-ics and 'Rpt' gaia; I will be moving them all under the same section eventually (Bug 1159345) but if you could please hide them all that'd be great.

Comment 4

[Robert Wood [:rwood]]

Update - all the raptor tasks are now reporting in the 'Rpt' treeherder group under 'gaia opt'

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Hidden.

Comment 6

[Robert Wood [:rwood]]

We changed the worker type for the raptor tests, but now they are not hidden anymore on treeherder :(

https://treeherder.mozilla.org/#/jobs?repo=gaia

The Rpt group is appearing at the top (in the raptor opt section).

Ryan (or another sheriff) could you please update so the raptor tasks are hidden on treeherder again please? Thanks very much!

Comment 7

[Ryan VanderMeulen [:RyanVM]]

I'm trying but it's not taking for some reason. Any idea what's going on, Cam?

Comment 8

[Cameron Dawson [:camd]]

Investigating now.  I checked the settings and they all LOOK right.  But I can see that the exclusion profile does NOT contain the signatures it should.  

Ahh, so there are more than 1 job types using the same symbol "se2".  And we're getting confused there and using the wrong job signature for it.  So we're "hiding" the wrong thing.  

I think we should be able to handle this case.  But it's on data ingestion as well as the exclusion profile generation.  Saving the job is getting the wrong signature.  And Adding the signature to the exclusion list is ALSO getting the wrong signature.

Fwiw: these are the three job types that have the same symbol:

Raptor Launch Test
Settings Launch Test (on base + pr)
Settings Launch Test (on base -1 commit)

Comment 9

[Cameron Dawson [:camd]]

Ryan - Would you consider this a candidate for your Top-5 list?

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Gaia's not a tree we monitor, so no. Can't speak for those who do look at it, though.

Comment 11

[Cameron Dawson [:camd]]

Ok.  Well, I'll keep it on my radar, because this could potentially show in other branches, too.  I'm not certain about that, but possibly.

Comment 12

[Robert Wood [:rwood]]

There is no need to hide the Raptor tasks anymore, as we are very close to having raptor ready to be 'live' anyway.

Comment 13

[Robert Wood [:rwood]]

Reopened, as we wish to hide the raptor tasks again on treeherder (too much results variance causing false alarms).

Comment 14

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/30632

Temporary work-around (as hiding is blocked at the moment); just don't post the raptor-tester and raptor-results tasks on treeherder. Tests will still run, and can be looked at via the raptor decision task which will still be visible.

Comment 15

[Robert Wood [:rwood]]

Thanks, landed work-around:

https://github.com/mozilla-b2g/gaia/commit/e7166cc72227de2367baccccb732640152aa2a48

Comment 16

[Robert Wood [:rwood]]

The Raptor task names are now changed and reporting to treeherder production (gaia and gaia-master) as unique names only:

Raptor(cal clk con dlr eml fm gal mus set sms tst vid)

i.e. https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=0f13023ab56c3276a5542121c259684204041a39

Can all of the Raptor tasks please be hidden on treeherder ASAP please? Thanks :camd, appreciated!

Comment 17

[Cameron Dawson [:camd]]

OK, the tests mentioned in comment 16 are now hidden.  Please let me know if you need anything else hidden that I missed.

Comment 18

[Robert Wood [:rwood]]

Looks awesome, thanks Cameron!