Closed Bug 1196933 Opened 5 years ago Closed 2 years ago

Add a Treeherder API to get jobs and related buildernames, codes, shorthand names, etc.

Categories

(Tree Management :: Treeherder: API, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: jmaher, Unassigned)

References

Details

Attachments

(1 file)

so many tools hack platform names, test names, branch names, options, and job codes- usually from a buildername.

It would be nice as we have code that uses shorthand names to query a service and get the buildername or another name for something.

Say we have:
bc1- this should be able to translate to mochitest-browser-chrome-1

For SETA, this would simplify a lot of code as we could depend on this module and then query the SETA API to do the translations for us.  It would simplify:
13:28 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/static/js/seta.js#L73
13:30 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/src/server.py#L373
13:30 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/src/server.py#L507
13:31 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/dailyjobs_query.py#L70 <- ignores builds/valgrind
13:32 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/failures.py#L51 <- output
13:33 <jmaher|mtg> https://github.com/dminor/ouija/blob/master/tools/seta.py#L131 <- b2g

alert manager, talos (compare.py), and alert bot would gain immediate benefit from this.
I think this should use allthethings.json or something like that as a master source.
Blocks: 1196934
Blocks: 1196936
I recently had to do the buildername -> TH symbols translation, and I used functions from TH's buildbot.py https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py#L972
maybe we could do this instead:

wget https://treeherder.mozilla.org/api/jobnames/

{'mozilla-inbound': [
  {'windows 8 x64 debug mochitest mochitest other m(oth)': {
    'jobname': 'Windows 8 64-bit mozilla-inbound debug test mochitest-other',
    'signature': 'c754fca95be6c3999ef187e358b832300826590f',
    'group': 'mochitest',
    'groupcode': 'M',
    'name': 'mochitest-other',
    'code': 'oth',
    'platform': 'Windows 8 64-bit',
    'buildtype': 'debug'
  },
  {...}
  ]
}
Summary: create a python package which handles mapping buildernames -> all the other shorthand versions → Add a Treeherder API to get jobs and related buildernames, codes, shorthand names, etc.
Component: General → Treeherder: API
Product: Testing → Tree Management
Version: unspecified → ---
:camd, :jfrench: what do you think about this?
Flags: needinfo?(tojonmz)
Flags: needinfo?(cdawson)
:adusca, I forgot to add you, thanks for the reminder on irc- what do you think?
Flags: needinfo?(alicescarpa)
We might be able to use this for bug 1053203 for the UserGuide (or wherever some master table of Builds and Tests eventually lives).
Flags: needinfo?(tojonmz)
For bug 1194830 we are already creating the treherder.mozilla.org/api/project/{branch}/runnable_jobs endpoint that contains a list of all the current buildbot buildernames and their symbols on that branch. I believe that when it lands [1] the endpoint will be available to anyone to read from. This is what an entry for a job currently looks like:

{
            "build_system_type": "buildbot",
            "job_group_symbol": "T",
            "job_group_name": "Talos Performance",
            "platform_option": "pgo",
            "job_type_description": "fill me",
            "signature": "3508b6b5320de5575fbc9b814574528379ce3226",
            "result": "runnable",
            "ref_data_name": "Windows XP 32-bit mozilla-inbound pgo talos tp5o",
            "machine_platform_architecture": "x86",
            "job_type_id": 128,
            "build_platform": "windowsxp",
            "job_type_name": "Talos tp",
            "platform": "windowsxp",
            "state": "runnable",
            "machine_platform_id": 4,
            "build_os": "win",
            "option_collection_hash": "f69e1b00908837bf0550250abb1645014317e8ec",
            "job_type_symbol": "tp",
            "job_group_description": "fill me",
            "job_coalesced_to_guid": null,
            "machine_platform_os": "win",
            "device_id": 1,
            "build_architecture": "x86",
            "device_name": "unknown",
            "build_platform_id": 4,
            "job_group_id": 9
        },

I think it is possible to add any information that is on allthethings.json to this endpoint. The main problem is that right now its goal is to only have currently existing jobs.  This means that when a buildername disappears from buildbot it disappears from this endpoint. But I think it would be possible to go around that by creating a new dict key to represent if a job is active or not and changing the way we clean old data.

If you decide to use this endpoint for this bug, I don't know if it would be better to land it at the same time as bug 1194830 to avoid changing the db schema on production or if it is not a big deal. If you only need a couple of extra keys it might only take a couple of lines on my PR, I don't know if it would be appropriate to just put it on the current PR or not.

[1] https://github.com/mozilla/treeherder/pull/892
Flags: needinfo?(alicescarpa)
This is the current full response for api/project/mozilla-inbound/runnable_jobs/
:adusca, this is really great.  I believe that end point will work- I will use definitions like that temporarily until the PR is merged in.
I'm a little late chiming in, but yeah.  This sounds really awesome.  Very useful indeed! :)
Flags: needinfo?(cdawson)
See Also: → 1043249
See Also: → 1174186
Marking incomplete in favour of the "see also" section bugs, plus the existing APIs.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.