1213262 - Give uniqueness to the steps on the Buildbot logs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

In bug 1212500 we have certain steps that cannot be differentiate since the first two arguments of the command are the same.
If we give unique identifiers to each step by adding a description to each we should be able to.

========= Started 'rm -rf ...' (results: 0, elapsed: 0 secs) (at 2015-10-09 04:12:23.350153) =========
========= Started 'bash -c ...' (results: 0, elapsed: 19 secs) (at 2015-10-09 04:12:23.462656) =========
========= Started 'rm -rf ...' (results: 0, elapsed: 1 secs) (at 2015-10-09 04:12:42.895826) =========
========= Started 'bash -c ...' (results: 0, elapsed: 30 secs) (at 2015-10-09 04:12:44.459870) =========

Comment 1

[Kyle Lahnakoski [:ekyle]]

Can we see the remainder of the command in the following lines?  jgraham mentioned he has the same problem with lines like:

> ========= Started 'c:/mozilla-build/python27/python -u ...'

If so, I can look into parsing that too.  By parsing it we can look at old records too.

The ETL does not care what text is between "Started " and "(results", if you add uid, or descriptions, they will be picked up, and you can report them.

Comment 2

[Armen [:armenzg]]

+jgraham, nthomas

If we pass a description='blah' paramater it should be sufficient.

Do you need the line to show the actual command? (e.g. Android and desktop builds would have a differentiating Buildbot step) or would a known identifier be sufficient or preferred? (e.g. Android and desktop jobs have a "run_harness" step)

The latter is much easier.

For the former, this is what it would be required.

The current text comes from getText() in the log uploader [1]
getText() calls describe() and I believe this is the one we care about:
> return ["'%s" % words[0], "%s" % words[1], "...'"]

I think adding a third case would be useful, however, this would require deploying a new version of the Buildbot masters.

[1] http://hg.mozilla.org/build/buildbotcustom/file/default/bin/log_uploader.py#l166
[2] http://mxr.mozilla.org/build/source/buildbot/master/buildbot/steps/shell.py#151

Comment 3

[Armen [:armenzg]]

<armenzg> ekyle, do you want anyone working on this? or fine to remove dependency? https://bugzilla.mozilla.org/show_bug.cgi?id=1213262
<ekyle> armenzg: those steps are already refined further by looking at the ?next? line
<ekyle> Just adding more values of the sequence does not do a good job; the parser already parses the most popular command lines
<armenzg> ekyle, so it's fine to close as INVALID?
<ekyle> armenzg: close as FIXED
<ekyle> armenzg: I will do that with some pointers

Comment 4

[Kyle Lahnakoski [:ekyle]]

The parse_builder_message()[1] is responsible for parsing the command lines found in the text logs.  Only the most common cases are covered here.

[1] https://github.com/klahnakoski/TestLog-ETL/blob/8530de86e8e038f961a6864efcf65f20022c09ac/testlog_etl/transforms/pulse_block_to_job_logs.py#L261