Closed Bug 918890 Opened 7 years ago Closed 6 years ago

B2G mozharness jobs should always save logcat even on success

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(1 file)

It might be useful to always have access to a logcat even if the job succeeded. Off the top of my head, we might want to:

* compare logcat of a known good run
* debug issues that don't trip up tbpl log parsing
* extract statistical data out of them (?)

The problem is that logcat can be very long, and the tbpl logs already take awhile to open as is. Therefore I propose that we upload a separate logcat file alongside the normal log (this depends on bug 749421).

I think we would still want to have logcat of failing jobs in the normal logs, but here I think it would be better if we could just dump a summary that includes the lines containing errors instead of the whole thing. That way we'd be able to detect anomalies in the logcat easily, and then open up the separate file to do further debugging if necessary.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
I got rid of the _dump_logcat function because it was doing more than the name implied and when I pared it down so it is consistent with its name, I found there were only two lines of code in it (and it only gets called in one place).
Attachment #8343753 - Flags: review?(ato)
Attachment #8343753 - Flags: review?(ato) → review+
in production
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
And yet this is actually busted!

08:14:19     INFO - Copying /builds/slave/test/build/emulator-5554.log to /builds/slave/test/build/blobber_upload_dir
08:14:19    ERROR - Can't copy /builds/slave/test/build/emulator-5554.log to /builds/slave/test/build/blobber_upload_dir: [Errno 21] Is a directory: '/builds/slave/test/build/blobber_upload_dir'!

see, e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=31748267&tree=Mozilla-Inbound#error2 for the above log snippet

I also wonder if we should have had releng review here. Or what. [though admittedly I doubt I would have caught this problem]
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Strange.. it's working fine on Ash. I must have forgotten to qref the patch or something. Investigating.
Yep, I had a fix in ash-mozharness that I evidently forgot to update in mozharness. Sorry about that, pushed a bustage fix:
https://hg.mozilla.org/build/mozharness/rev/a8310716f594

(In reply to Justin Wood (:Callek) from comment #4)
> I also wonder if we should have had releng review here. Or what. [though
> admittedly I doubt I would have caught this problem]

Fwiw, we ask for releng review whenever modifying build configs or mozharness itself. The test scripts are generally written and owned by the a-team (though we do occasionally ask for releng review for those too). This was just a case of me making a stupid mistake.
Yeah. FTR, I've verbally given :ahal and :jgriffin the ok to write and review their own test-only mozharness patches, to reduce Releng bottleneck.
Duplicate of this bug: 948296
Duplicate of this bug: 949340
In production
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.