1076686 - treeherder: Log parser should pick up error messages

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[James Lal [:lightsofapollo]]

By default we use live logs everywhere which will unlikely be suitable for treeherder logging... We need to ensure the _backing__ log is sent (or we send the live log after and ensure treeherder handles redirects) and log parsing is functional

Comment 1

[James Lal [:lightsofapollo]]

This only works with gzipped logs right now so we can easily work around it by fixing 1073092

Comment 2

[James Lal [:lightsofapollo]]

Attachment: https://github.com/taskcluster/docker-worker/pull/48

Note: The livelog server does not compress on the fly (this would probably work very poorly and/or result in terrible compression) this only effects the "backing: log which we store (and pay for) on s3.

Comment 3

[James Lal [:lightsofapollo]]

Comment on attachment 8500150 [details] [review]
https://github.com/taskcluster/docker-worker/pull/48

Flagged on the wrong bug this is for gzip (moved to the correct bug now)

Comment 4

[James Lal [:lightsofapollo]]

Gzipped logs help but we still don't have test failures showing correctly so more debugging is needed... I think we should just try to appease the parser rather then updating it for this bug.

Comment 5

[James Lal [:lightsofapollo]]

The bug here is really we don't know exactly what tricks to use to ensure treeherder picks up on our assertion messages ... We are using the correct TBPL format across the gaia tests but still do not see the assertions or the bug associations.

Comment 6

[James Lal [:lightsofapollo]]

Okay so this one is fairly easy (to work around) basically you just need to appease the treeherder 'StepParser' by being inside of a buildbot step https://github.com/mozilla/treeherder-service/blob/d8c423f92d021517d6fff47f9c97cf63f755ca79/treeherder/log_parser/parsers.pyx#L139.

So anything we want full parsing support basically just needs to wrap the logs in something like this:

========= Started test run (results: 0, elapsed: 0 secs) (at 2013-06-05 12:51:48.767751) =========
<INSERT LOGS HERE>
========= Finished test run (results: 0, elapsed: 60 secs) (at 2013-06-05 12:51:48.767751) =========

I am going to move this to a gaia issues as this is easy to add at the "entrypoint" level so all tests/runs/builds have at least this one step.

Comment 7

[James Lal [:lightsofapollo]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/25052

Comment 8

[James Lal [:lightsofapollo]]

UI looks like this: https://treeherder.allizom.org/ui/#/jobs?repo=gaia&revision=664cde9371cc039d87bd69a899944d6261cddaa0

cc: bc/davehunt see https://github.com/lightsofapollo/gaia/blob/bug-1076686/build/docker/gaia-taskenv/bin/buildbot_step as long as you have a buildbot step wrapper treeherder (today) will work just fine and parse out the bugs/exceptions/etc.. The format is very specific though.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to James Lal [:lightsofapollo] from comment #8)
> UI looks like this:
> https://treeherder.allizom.org/ui/#/
> jobs?repo=gaia&revision=664cde9371cc039d87bd69a899944d6261cddaa0
> 
> cc: bc/davehunt see
> https://github.com/lightsofapollo/gaia/blob/bug-1076686/build/docker/gaia-
> taskenv/bin/buildbot_step as long as you have a buildbot step wrapper
> treeherder (today) will work just fine and parse out the
> bugs/exceptions/etc.. The format is very specific though.

If we add this to the Python test runner wouldn't it mess up the current buildbot tests using the runner by adding more steps? Also, looking at your first link, I found a parsed log with steps that were all green despite them including an exception.

Comment 10

[James Lal [:lightsofapollo]]

We don't need to take any steps for gaia (this will "just work" in gaia-try and now in TC) but for device runs we should wrap the jenkins output with something like this ( I also want to begin the conversation of moving all our device testing into TC but that is a much longer thing ).

Can you point out which failure was missed? The log parser seems to have a pretty good list of what we consider errors.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to James Lal [:lightsofapollo] from comment #10)
> We don't need to take any steps for gaia (this will "just work" in gaia-try
> and now in TC) but for device runs we should wrap the jenkins output with
> something like this ( I also want to begin the conversation of moving all
> our device testing into TC but that is a much longer thing ).

Our device testing uses the same harness as our desktop testing, so adding any of these build steps will affect both. I can see if I can make this output dependent on the target (most likely blocked by bug 1038868) but it feels like a rather ugly hack.

> Can you point out which failure was missed? The log parser seems to have a
> pretty good list of what we consider errors.

https://treeherder.allizom.org/ui/logviewer.html#?job_id=73835&repo=gaia indicates a successful step. Screenshot: https://www.dropbox.com/s/xv9v0r21qs3bna4/Screenshot%202014-10-14%2011.35.12.png?dl=0 compared to what I'd expect to see: https://www.dropbox.com/s/wksja3nft238umm/Screenshot%202014-10-14%2011.36.23.png?dl=0

Comment 12

[Kevin Grandon :kgrandon]

Comment on attachment 8503669 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/25052

I don't mind reviewing and iterating on this, but it seems Dave might know more about this than I do, and the screenshots in comment 11 seem wrong. Let me know what you want to do.

Comment 13

[James Lal [:lightsofapollo]]

Fixed as part of https://github.com/mozilla-b2g/gaia/commit/fd7aaa083530304ad1e4706eb0020d43f866eed1 with dave's comment addressed