Closed Bug 601216 Opened 11 years ago Closed 10 years ago

Create buildbot logparser for generating data for waronorange hbase system

Categories

(Tree Management Graveyard :: OrangeFactor, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Unassigned)

References

Details

I've implemented a log parser for the buildbot logs produced by our test runs.
It scans these logs and produces a line of output in JSON format, which contains
the kinds of data we think we could be useful in analyzing random oranges
and other factors.

The exact format of the output is described at 
http://hg.mozilla.org/automation/logparser/file/d7dfabcc25b8/logparser/logparser.py#l549
and
http://hg.mozilla.org/automation/logparser/file/d7dfabcc25b8/logparser/logparser.py#l139

Examples are at http://hg.mozilla.org/automation/logparser/file/tip/logparser/tests

Ideally, we'd want some additional data that isn't in the raw buildbot logs,
but is in the processed tinderbox logs.  This info includes OS version and
starttime.  If we're going to use this with processed tinderbox logs
instead of raw buildbot logs, I'll have to update the script to get that
additional info; otherwise we'll have to figure out an alternate mechanism
to get this data.

As far as I've been able to determine, the life cycle of a log is this:

1 - test machine produces raw buildbot log, and uploads it to FTP server
2 - test machine e-mails raw buildbot log to tinderbox, along with some
extra data that isn't in the raw log; this extra data looks like this:

tinderbox: tree: Firefox
tinderbox: builddate: 1285956707
tinderbox: status: testfailed
tinderbox: build: Rev3 Fedora 12 mozilla-central debug test mochitest-other
tinderbox: errorparser: unittest
tinderbox: binaryurl: 
tinderbox: logcompression: gzip
tinderbox: logencoding: base64
tinderbox: END

3 - tinderbox tracks comments for failed test cases in a separate file, e.g.
http://tinderbox.mozilla.org/Firefox/notes.txt
4 - tinderbox creates status pages based on 2) and 3)

It isn't clear to me at what point in this cycle this logparser will 
operate; I'm currently assuming it's on the raw buildbot logs from step 1).
I neglected to mention that the parser lives at http://hg.mozilla.org/automation/logparser/
This has been functioning for quite a while now.
Assignee: jgriffin → nobody
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Component: General → Orange Factor
QA Contact: general → orangefactor
Resolution: --- → FIXED
Product: Testing → Tree Management
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.