1196933 - Add a Treeherder API to get jobs and related buildernames, codes, shorthand names, etc.

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder: API, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

so many tools hack platform names, test names, branch names, options, and job codes- usually from a buildername.

It would be nice as we have code that uses shorthand names to query a service and get the buildername or another name for something.

Say we have:
bc1- this should be able to translate to mochitest-browser-chrome-1

For SETA, this would simplify a lot of code as we could depend on this module and then query the SETA API to do the translations for us.  It would simplify:
13:28 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/static/js/seta.js#L73
13:30 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/src/server.py#L373
13:30 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/src/server.py#L507
13:31 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/dailyjobs_query.py#L70 <- ignores builds/valgrind
13:32 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/failures.py#L51 <- output
13:33 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/seta.py#L131 <- b2g

alert manager, talos (compare.py), and alert bot would gain immediate benefit from this.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I think this should use allthethings.json or something like that as a master source.

Comment 2

[Alice Scarpa [:adusca]]

I recently had to do the buildername -> TH symbols translation, and I used functions from TH's buildbot.py https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py#L972

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

maybe we could do this instead:

wget https://treeherder.mozilla.org/api/jobnames/

{'mozilla-inbound': [
  {'windows 8 x64 debug mochitest mochitest other m(oth)': {
    'jobname': 'Windows 8 64-bit mozilla-inbound debug test mochitest-other',
    'signature': 'c754fca95be6c3999ef187e358b832300826590f',
    'group': 'mochitest',
    'groupcode': 'M',
    'name': 'mochitest-other',
    'code': 'oth',
    'platform': 'Windows 8 64-bit',
    'buildtype': 'debug'
  },
  {...}
  ]
}

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

:camd, :jfrench: what do you think about this?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

:adusca, I forgot to add you, thanks for the reminder on irc- what do you think?

Comment 6

[Jonathan French (:jfrench)]

We might be able to use this for bug 1053203 for the UserGuide (or wherever some master table of Builds and Tests eventually lives).

Comment 7

[Alice Scarpa [:adusca]]

For bug 1194830 we are already creating the treherder.mozilla.org/api/project/{branch}/runnable_jobs endpoint that contains a list of all the current buildbot buildernames and their symbols on that branch. I believe that when it lands [1] the endpoint will be available to anyone to read from. This is what an entry for a job currently looks like:

{
            "build_system_type": "buildbot",
            "job_group_symbol": "T",
            "job_group_name": "Talos Performance",
            "platform_option": "pgo",
            "job_type_description": "fill me",
            "signature": "3508b6b5320de5575fbc9b814574528379ce3226",
            "result": "runnable",
            "ref_data_name": "Windows XP 32-bit mozilla-inbound pgo talos tp5o",
            "machine_platform_architecture": "x86",
            "job_type_id": 128,
            "build_platform": "windowsxp",
            "job_type_name": "Talos tp",
            "platform": "windowsxp",
            "state": "runnable",
            "machine_platform_id": 4,
            "build_os": "win",
            "option_collection_hash": "f69e1b00908837bf0550250abb1645014317e8ec",
            "job_type_symbol": "tp",
            "job_group_description": "fill me",
            "job_coalesced_to_guid": null,
            "machine_platform_os": "win",
            "device_id": 1,
            "build_architecture": "x86",
            "device_name": "unknown",
            "build_platform_id": 4,
            "job_group_id": 9
        },

I think it is possible to add any information that is on allthethings.json to this endpoint. The main problem is that right now its goal is to only have currently existing jobs.  This means that when a buildername disappears from buildbot it disappears from this endpoint. But I think it would be possible to go around that by creating a new dict key to represent if a job is active or not and changing the way we clean old data.

If you decide to use this endpoint for this bug, I don't know if it would be better to land it at the same time as bug 1194830 to avoid changing the db schema on production or if it is not a big deal. If you only need a couple of extra keys it might only take a couple of lines on my PR, I don't know if it would be appropriate to just put it on the current PR or not.

[1] https://github.com/mozilla/treeherder/pull/892

Comment 8

[Alice Scarpa [:adusca]]

Attachment: api/project/mozilla-inbound/runnable_jobs/ response

This is the current full response for api/project/mozilla-inbound/runnable_jobs/

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

:adusca, this is really great.  I believe that end point will work- I will use definitions like that temporarily until the PR is merged in.

Comment 10

[Cameron Dawson [:camd]]

I'm a little late chiming in, but yeah.  This sounds really awesome.  Very useful indeed! :)

Comment 11

[Ed Morley [:emorley]]

Marking incomplete in favour of the "see also" section bugs, plus the existing APIs.