1062827 - Change names used for job results (eg "busted", "testfailed") to be closer to those used by TaskCluster than buildbot

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

Currently treeherder uses the same names to describe the result states as TBPL did:

https://github.com/mozilla/treeherder-service/blob/bf60d1908e85f925cdf7a7a9b5bdc52082e64bab/treeherder/etl/buildbot.py#L3
RESULT_DICT = {
    0: "success",
    1: "testfailed",
    2: "busted",
    3: "skipped",
    4: "exception",
    5: "retry",
    6: "usercancel"
}

These names seem to be somewhat arbitrary, and do not even match what buildbot calls them:

http://hg.mozilla.org/build/buildbot/file/4b5a85b140b3/master/buildbot/status/builder.py#l25
    25 SUCCESS, WARNINGS, FAILURE, SKIPPED, EXCEPTION, RETRY, CANCELLED = range(7)
    26 Results = ["success", "warnings", "failure", "skipped", "exception",
    27            "retry", "cancelled"]

Given they seem to have been arbitrarily picked & that names like "testfailed" don't really describe build job types well + "busted" is just not very descriptive - I think we should rename them to something like:

success  ->  <no change> (or "pass"?)
testfailed  ->  warnings
busted  ->  failed
skipped  ->  <no change>
exception  ->  infra-failure
retry  ->  <no change>
usercancel  ->  <no change>

This will require quite a few changes throughout the service but particularly the UI, so will be a bit of a pain - but I think it will be best if we change now, rather than end up being stuck with these names X years down the line when we're not using buildbot and our infra has changed.

Comment 1

[Jonathan French (:jfrench)]

Checking with Wes and Ryan to confirm they're happy also with the name-mapping changes above? eg. we might have enough room for the full, 'infrastructure-failure' as a status, dunno.

I'll have a look at treeherder-ui to start.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

fine with me

Comment 3

[Jonathan French (:jfrench)]

I've pushed a wip change to my own remote, 'update-job-result-names' for the UI. I'll hold off on opening a PR for now, pending feedback from Wes also.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Fine with me as well.

Comment 5

[Jonathan French (:jfrench)]

Attachment: treeherder-ui-PR#248

Opened preliminary PR for the UI side.

Comment 6

[Jonathan French (:jfrench)]

Given this change hasn't bubbled up in channel as a high priority since I opened the -ui PR 5 months ago, I'd like to close my PR for now, if that's ok Ed? We can always refer to it, and assuming the code is still relevant, re-open the PR when we get to the bug.

I figure that way it doesn't clutter the Open PR list.

Comment 7

[Ed Morley [:emorley]]

Yeah sounds sensible for now :-)

Comment 8

[Jonathan French (:jfrench)]

Cool, done.

Comment 9

[Ed Morley [:emorley]]

This will be easier after bug 1395254.

Comment 10

[Ed Morley [:emorley]]

There may also be extra states from TaskCluster that we currently lump into one, that we could instead separate out for improved UX.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

cleaning up old bugs