522792 - tinderbox log highlighter should highlight which build steps failed

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I believe there's a list of regular expressions somewhere that controls which lines get highlighted at the top of a tinderbox log.

I think it would make many logs *much* more comprehensible if the logs highlighted the messages:

... failed
... warnings

I'm not sure whether you can use multi-line regexes or not; if you can, it should be trivial:  just highlight any line ending in " failed" or " warnings" that's one line after the line "======== BuildStep started ========".

If not, I think it's worth tweaking the code that converts the buildbot logs to tinderbox logs so that these messages can be highlighted; this would help people understand all the weird errors by highlighting the part of the build process that had the failure.

Comment 1

[Armen [:armenzg]]

Bug 516443 is to be considered as well.

Comment 2

[(not currently active) Ted Mielczarek]

http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/ep_unix.pl
and
http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/ep_unittest.pl

Contain the regexes that are used for normal builds and unittest builds, respectively.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

This sounds like a good idea,  I am going to move to future until someone has time to work on this.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This is pretty simple to do.  It just requires a simple modification to the printing code added in:
http://hg.mozilla.org/build/buildbot/rev/f1fda48634d6
and then changes to the files ted mentioned above.  I'll post the patches that should be necessary shortly, but I can't test them.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: untested buildbot patch

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: untested tinderbox patch

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This would really make lots of obscure failures easier to diagnose from the logs.  Any chance I could talk somebody into working on this?

Comment 8

[(not currently active) Ted Mielczarek]

I've hacked the error parser without ever actually setting up a tinderbox. I just have a "test-errorparser.pl" that contains:
require 'ep_unittest.pl';

while(<>) {
    if (has_error($_)) {
        print;
    }
}

I stick that in the webtools/tinderbox dir, run it, and paste in log lines, and it will echo back lines that match the error regex. If you're comfortable with that, then get review from cls, and that's all that's necessary.

Comment 9

[Daniel Holbert [:dholbert]]

Here's a failure log from today where we could've used this:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1259886360.1259887933.11523.gz

The failure was this, during crashtests:
{
REFTEST TEST-PASS | file:///builds/moz2_slave/mozilla-central-macosx-opt-unittest-everythingelse/build/reftest/tests/dom/base/crashtests/371124-2.html | (LOAD ONLY)
process killed by signal 30
program finished with exit code -1
elapsedTime=21.912001
TinderboxPrint: crashtest<br/><em class="testfail">T-FAIL</em>
Unknown Error: command finished with exit code: -1
=== Output ended ===
======== BuildStep ended ========
}

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

instead of making buildbot-side changes, could you a regex like "program finished with exit code [^0]"

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #10)
> instead of making buildbot-side changes, could you a regex like "program
> finished with exit code [^0]"

I want to catch *all* the cases of build step failure (does that really catch all of them), and I want it to highlight a useful description of what failed.

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And, to be clear:  the buildbot changes are just slightly modifying what's already one of our local patches to buildbot.

Comment 16

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 17

[Chris Cooper [:coop] (he/him)]

dbaron: would you still like this? 

We (releng) would like to stop propping up tinderbox in an effort to get people to switch to TBPL/BBPL. If we can accomplish the same thing with a buildbot-only patch, we'd certainly prefer that.

Comment 18

[Chris Cooper [:coop] (he/him)]

Based on the discussions at the "How do we kill tinderbox?" and "War on Orange" meetings at the all-hands, it looks like the Orange Factor tool will be replacing the tinderbox log store soon (matter of weeks?).

Let's file Orange Factor bugs to make sure log parsing meets our current requirements.