489524 - should have a talos tinderbox error parser

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect, P3)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

In bug 474950 comment 3, ted wrote:

Ideally, we could get the tinderbox error parser to highlight the relevant
failure lines. If need be, we can make an ep_talos.pl to highlight
Talos-specific errors. If they show up in the summary, then it's fairly clear
what happened.

The unit test error parser looks like this:
http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/ep_unittest.pl

Comment 1

[(not currently active) Ted Mielczarek]

I think this is probably the only sane solution to the Talos log problem. If people see something in the log summary, they will pay attention to that and not the bogus disconnection message.

Comment 2

[Jesse Ruderman]

Getting rid of the bogus disconnection message would be nice too ;)

Comment 3

[alice nodelman [:alice] [:anode]]

Moving to Future till we can dedicate time to this.

Comment 4

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Moving this to Testing:Talos so it can be evaluated.

Comment 7

[alice nodelman [:alice] [:anode]]

Are we still wanting a tinderbox log parser?  Seems that we are quickly moving everything off tinderbox entirely for dashboards...

Comment 8

[Phil Ringnalda (:philor)]

I'm unable to speak to the wonders of whatever is going to replace tinderbox for storing and serving logs, since years of working on Mozilla stuff has taught me to treat all such promised things that are going to solve all our problems as unicorn-ponies dancing on a rock candy mountain until there's a bug with an 80% patch attached. If there is, I'd be happy to look at it, and say whether it would replace the current need.

An errorparser does three things for you:

* it puts the actual lines from the log that constituted a failure up at the top of the log, linked so you can find them in context
** as a result, tbpl can tell what the error was, can put it in the summary popup that opens when you click on a failing build, and with maybe some tweaking to deal with talos failures maybe not being as strongly associated with test filenames as they are in other suites, could suggest bugs to automatically comment in and star the failure with

* it makes the brief log actually brief, trimming off all the junk and only leaving the context around the failure, so instead of being faced with a whole long log with dozens of steps spewing their environment and outputting nothing or an enormous list of files being unzipped, the log actually shows what went wrong

* it makes the suite author restrict their output to something that can be found - I don't even remember whether there are things other than "FAIL" that mark talos failures, but I do know that the TinderboxPrinting isn't working well. There is currently one fairly common failure on TraceMonkey in dromaeo_sunspider, which I thought for most of the time I was ignoring it was actually two separate failures happening half as often, since the TinderboxPrints were telling me that it was a crash on Linux and a hang on Windows

Those things together are why nearly every bug I file about talos failures, a few months later you ask "is this still happening?" and if I'm in an honest mood I say "nobody has the slightest idea, because talos is annoying to star so everyone just ignores failures in it." In the place where we look for every single other sort of failure to see what happened, with a talos failure it just says "Summary is empty." Unless someone is really bored or really feeling guilty about having maybe caused it, we just figure that means it's a mysterious orange with no output, and ignore it.

Comment 9

[Jeff Hammel]

This bug no longer seems applicable.  TBPL effectively replaced this.  There has been an ongoing concerted effort to make talos error parsing there sane