601216 - Create buildbot logparser for generating data for waronorange hbase system

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Jonathan Griffin (:jgriffin)]

I've implemented a log parser for the buildbot logs produced by our test runs.
It scans these logs and produces a line of output in JSON format, which contains
the kinds of data we think we could be useful in analyzing random oranges
and other factors.

The exact format of the output is described at 
http://hg.mozilla.org/automation/logparser/file/d7dfabcc25b8/logparser/logparser.py#l549
and
http://hg.mozilla.org/automation/logparser/file/d7dfabcc25b8/logparser/logparser.py#l139

Examples are at http://hg.mozilla.org/automation/logparser/file/tip/logparser/tests

Ideally, we'd want some additional data that isn't in the raw buildbot logs,
but is in the processed tinderbox logs.  This info includes OS version and
starttime.  If we're going to use this with processed tinderbox logs
instead of raw buildbot logs, I'll have to update the script to get that
additional info; otherwise we'll have to figure out an alternate mechanism
to get this data.

As far as I've been able to determine, the life cycle of a log is this:

1 - test machine produces raw buildbot log, and uploads it to FTP server
2 - test machine e-mails raw buildbot log to tinderbox, along with some
extra data that isn't in the raw log; this extra data looks like this:

tinderbox: tree: Firefox
tinderbox: builddate: 1285956707
tinderbox: status: testfailed
tinderbox: build: Rev3 Fedora 12 mozilla-central debug test mochitest-other
tinderbox: errorparser: unittest
tinderbox: binaryurl: 
tinderbox: logcompression: gzip
tinderbox: logencoding: base64
tinderbox: END

3 - tinderbox tracks comments for failed test cases in a separate file, e.g.
http://tinderbox.mozilla.org/Firefox/notes.txt
4 - tinderbox creates status pages based on 2) and 3)

It isn't clear to me at what point in this cycle this logparser will 
operate; I'm currently assuming it's on the raw buildbot logs from step 1).

Comment 1

[Jonathan Griffin (:jgriffin)]

I neglected to mention that the parser lives at http://hg.mozilla.org/automation/logparser/

Comment 2

[Jonathan Griffin (:jgriffin)]

This has been functioning for quite a while now.