1270627 - fix missing builder names in treeherder for release promotion jobs

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Kim Moir [:kmoir] ET]

For example, the jobs with ? means the job isn't mapped to anything in the treeherder code

https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=0b8492c110bec959b94e3d54d5bd5ca7f7f97f6c

The code for this is in https://github.com/mozilla/treeherder

in  treeherder/etl and  tests/etl

Comment 1

[Kim Moir [:kmoir] ET]

Notes from talking to rail

you can see that some job names are fixed here.
https://treeherder.allizom.org/#/jobs?repo=mozilla-beta
(staging treeherder)

in release tasks, the treeherder jobs are defined as staging or production
in the templates.  For example, for the beetmover task
https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/firefox/beetmove_image.yml.tmpl
you can see 
treeherderEnv:
                - staging

and 
 routes:
            - tc-treeherder-stage.{{ branch }}.{{ revision_hash }}

We can make a pull request to change these to production as a first step.  Then we can deploy once a new treeherder is deployed to production.

Comment 2

[Kim Moir [:kmoir] ET]

Looks good here with our latest beta in release promotion (Firefox 47.0b5)

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=93a53170dedffdff45bf9eb8dac6e5ef7a13c4ba

I think the next step is to add the remaining jobs indicated with question marks in treeherder etc.

Comment 3

[Kim Moir [:kmoir] ET]

So you can see there are jobs with question marks here

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=7d1f3450acc47025876964c1eca854ae027934f3

release-mozilla-beta-fir opt 	
release-mozilla-beta_fir opt

They'll need to be added to treeherder so they appear properly. There are a variety of job types, not sure if it important to separate them into specific headings are just include them as a generic release promotion job.

Comment 4

[Kim Moir [:kmoir] ET]

Here is an example of a previous change that updates the tests too in treeherder 
https://bugzilla.mozilla.org/show_bug.cgi?id=1212177#c10

Comment 5

[Francis Kang]

Attachment: The buildernames for the missing jobs have no platform information inside. I tried to group it into linux64, but the regex matches jobs from other platforms as well.

Comment 6

[GitHub Autolander Bot]

Attachment: [treeherder] hfkang:bug1260627-pr > mozilla:master

Comment 7

[Francis Kang]

Attachment: The buildernames for the missing jobs have no platform data, and adding that will require changing the task templates to include platform info

Comment 8

[Francis Kang]

Should we update the task templates to include test platform information?

Comment 9

[Rail Aliiev [:rail]]

I just checked the task templates and it looks like we don't explicitly submit most of those (except uptake monitoring I think), they come via buildbot. https://github.com/mozilla/releasetasks/search?p=2&q=treeherder&utf8=%E2%9C%93

Sounds like treeherder tries to guess the platform and falls back to something (first N characters?).

Comment 10

[Ed Morley [:emorley]]

Moving to Treeherder since the PR is for the Treeherder repo. Have left some comments on the PR - please re-request review on the PR-link attachment here once ready for re-review :-)

Comment 12

[Ed Morley [:emorley]]

Comment on attachment 8757999 [details] [review]
[treeherder] hfkang:bug1260627-pr > mozilla:master

Left some more comments in response to the platform change :-)

Comment 13

[Ed Morley [:emorley]]

Comment on attachment 8757999 [details] [review]
[treeherder] hfkang:bug1260627-pr > mozilla:master

Sorry for the delay. This looks good - if you could adjust the first line of the commit message of the deadcode removal commit to mention the deadcode removal (and rebase on master if possible), I'll merge this for you :-)

Comment 14

[Francis Kang]

I've rebased, and amended the commit :)

Comment 15

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0d9ae7a6b2b2555f700c53ffd2aa39b376f1260e
Bug 1270627 - Support release promotion jobs

These jobs have been added under the new Release group in the "other" platform:
 - Version Bump
 - Checksums Builder
 - Uptake Monitor
 - Updates
 - Bouncer Aliases
 - Bouncer Submission
 - Update Verify (moved from Updates group)

Since the jobs do not contain platform info in buildname, nor are
associated with a specific os platform, the regex captures all
of the jobs individually to assign it the "other" os_platform.

https://github.com/mozilla/treeherder/commit/3b2d48d319380e500e43812f7fecbf28c861fb1a
Bug 1270627 - Remove unused job_type dictionary

Remove JOB_TYPE_BUILDERNAME and extract_job_type(), as neither are used
anywhere.
The relevant test components have also been removed from test_buildbot.py.

Comment 16

[Ed Morley [:emorley]]

This should be deployed sometime this week :-)