1121837 - Reftest & crashtest timeouts & crashes are displayed as orange on Windows, despite being red on Linux/Mac

Status: NEW

(Testing :: Reftest, defect)

Description

[Daniel Holbert [:dholbert]]

I just pushed a Try run, which hit a crashtest hang, in a new crashtest that I added in the push:

  https://treeherder.mozilla.org/#/jobs?repo=try&revision=d13dd73a8a3f

As you can see, the hang turns the "C" jobs red on Mac & Linux, BUT, it turns them *orange* on Windows for some reason.

This inconsistency is strange & should probably be fixed.

(I'm filing this in TreeHerder, for now, but I suspect it probably really belongs elsewhere (maybe RelEng::[Something]), because I see the same inconsistency if I view the results via TBPL instead of treeherder.)

(FWIW, the hanging crashtest doesn't exist in the tree yet -- it's added in my push)

Comment 1

[Phil Ringnalda (:philor)]

Absolutely not treeherder, which displays a color based on the job status that buildbot has assigned.

Given that on Linux and Mac, the harness is returning "-9" and on Windows it's returning "1", I like blaming the harness before blaming buildbot.

Comment 2

[Daniel Holbert [:dholbert]]

It seems that *crashes* during Crashtest runs are similarly displayed as orange on Windows. (I'm 90% sure they're red on other platforms -- or at least, they should be, because they prevent a bunch of other tests [everything after the crash] from running.)

(This caused someone to mistake a crash for an intermittent-test-failure, over in bug 1124847 comment 9.)

Comment 3

[Daniel Holbert [:dholbert]]

(and the same seems to be true for reftests; crashes during reftests [and presumably hangs as well] turn the job orange on Windows, according to the Try run linked in bug 1124847 comment 9.)

(I guess this is unsurprising because the crashtest & reftest harnesses are basically the same)

Comment 4

[Phil Ringnalda (:philor)]

And mochitest, given that there's a crash in mochitest-chrome.

But, what does "red" mean? (Hint: if your answer doesn't wind up at "we should display 'red' and 'orange' and 'purple' as a single color, the color of failure" then you haven't finished thinking it through.)

Comment 5

[Daniel Holbert [:dholbert]]

(In reply to Phil Ringnalda (:philor) from comment #4)
> But, what does "red" mean?

IIRC, the distinction between red & orange was originally something like this:

* orange: something failed / went wrong, but we weren't blocked from running subsequent tests.
* red: something failed / went wrong, and we *were* blocked from running subsequent tests.

This is an important distinction; red is significantly worse than orange, because it indicates that we're *missing information* about whether a pile of other tests passed or failed.

(This is why build failures are red -- because they block you from running tests -- as are crashes, because we don't get to run any tests after the crash.)

I'd rather not deep-dive too much into coloring philosophy here, though -- regardless of the high-level significance, we can at least agree that the coloring is inconsistent right now, between Windows & other platforms.

Comment 6

[(not currently active) Ted Mielczarek]

I don't think it was ever that well-defined. We just abused buildbot's status codes and used red for FAILURE and orange for WARNINGS. We call WARNINGS "testfail" nowadays, I think, but i can't remember anyone ever proposing a real distinction here.

I think all non-infra (not going to rathole on what *that* distinction is) failures in test jobs should be orange.

Comment 7

[Daniel Holbert [:dholbert]]

Per end of comment 5, if we're going to (re)define the colors, we can do that in a spearate bug maybe -- if possible/easy, let's make this bug just scoped to fixing whatever's making us inconsistent between platforms.