816532 - Print logcat for failed robocop tests

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

Robocop tests collect and display logcat data at the end of a test run. When a test fails or crashes, the error is reported, then the next test is run...and only when all tests have completed is the logcat collected and printed in the log. Usually, the logcat only shows activity for the most recent (last) test. 

An example:

https://tbpl.mozilla.org/php/getParsedLog.php?id=17440427&full=1&branch=mozilla-inbound

It would be more useful if the logcat was dumped whenever an error is detected, or if the logcat for the entire run was collected.

Comment 1

[Ed Morley [:emorley]]

Attachment: Patch v1

Something like this?

This will always output the logcat/device info at the end of the entire run - and for robocop, will additionally output it after each individual runTests, if the exit code was non-zero. (I think dumping the logcat after every robocop runTests will be too verbose).

The only problem with this is if the non-zero return code happened on the last runTests, we'll end up dumping near-identical logcat twice. Does this matter?

Comment 2

[Geoff Brown [:gbrown]]

Comment on attachment 687790 [details] [diff] [review]
Patch v1

Review of attachment 687790 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks much Ed -- this is just what I had in mind. I would not worry about the double logcat on last test failure issue -- I don't think it will bother or confuse anyone.

Comment 3

[Ed Morley [:emorley]]

Thank you for the review :-)

https://hg.mozilla.org/integration/mozilla-inbound/rev/198a2cd21e1b

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/198a2cd21e1b

Comment 5

[Geoff Brown [:gbrown]]

This was a good fix, but I think we did not go far enough. We dump logcat when runTests returns an error, but robocop failures often do not return an error: we still see many robocop failures without logcats.

Comment 6

[Ed Morley [:emorley]]

I don't suppose you have some links?

The root cause is the harness not outputting the correct error code surely?

I guess we could always dump the logcat, but that's going to get bloated pretty quickly I think.

Comment 7

[Geoff Brown [:gbrown]]

Examples:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17851959&tree=Firefox&full=1#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=17881486&tree=Firefox&full=1#error0

I would prefer to dump the logcat only on error...I think the thing to do is tighten up our error detection.

Comment 8

[Ed Morley [:emorley]]

Since this is an orthogonal problem to the one fixed in this bug, re-closing in favour of the bug I've filed for this (bug 822823).