815676 - Add "Fatal signal 11 (SIGSEGV)" to mozharness b2g emulator tests' error list

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Andrew Halberstadt [:ahal]]

When there is a test failure we dump the logcat. I've found that often there are these lines:
11:42:45     INFO - F/libc    (  325): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1)

which usually means the b2g process has crashed. We should highlight when this happens to make triaging easier.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - Add error messages

I also added "[Javascript Error:" at the WARNING level since those types of messages may or may not indicate a problem.

Also note that the logcat only gets dumped when the parser has errors, so we shouldn't get tripped up by irrelevant seg faults in the logcat.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 685679 [details] [diff] [review]
Patch 1.0 - Add error messages

For the 'Fatal signal 11 (SIGSEGV)' error dict, could you add

'explanation': 'This usually means the b2g process has crashed'

or something similar to help people interpret the error?

Comment 3

[Andrew Halberstadt [:ahal]]

Added 'explanation' and pushed:
https://hg.mozilla.org/build/mozharness/rev/809c4cc296e8

Comment 4

[Andrew Halberstadt [:ahal]]

So it appears that these errors are not getting highlighted. Probably because we dump the logcat after the parser has already parsed the rest of the logs.

Comment 5

[Andrew Halberstadt [:ahal]]

Aki, do you know of some simple way to highlight errors after the parser has gone through the logs? Would we just have to re-parse it?

Comment 6

[Aki Sasaki (not active)]

I don't understand the problem here; do you have an example of what's happened, and what you want to happen?

Highlighted == "tbpl is showing them in the summary" or "these are getting marked as ERROR in the log" ?

Is the logcat part of the same run_command() ? Or a separate one?

Are you sure we're using the right error_list? ;-D

Comment 7

[Ed Morley [:emorley]]

TBPL will show them in the annotated summary as long as they are marked as ERROR in the log (they are still being marked as INFO).

Comment 8

[Jonathan Griffin (:jgriffin)]

The parser is invoked for the output of the test harness, but logcat isn't being dumped from the test harness.  So, we'd need another parser instance to run the logcat output through, in order to catch errors that show up there.

Comment 9

[Aki Sasaki (not active)]

Yeah, looks like we're just reading the logcat, and self.info()ing the contents:
http://hg.mozilla.org/build/mozharness/file/7f4515838fd9/scripts/b2g_emulator_unittest.py#l249

You can change that to a self.run_command("cat LOGCATFILE", error_list=APPROPRIATE_ERROR_LIST) to actually parse the contents for errors.

Comment 10

[Aki Sasaki (not active)]

Attachment: run logcat through run_command() to parse for errors

Comment 11

[Andrew Halberstadt [:ahal]]

Comment on attachment 688027 [details] [diff] [review]
run logcat through run_command() to parse for errors

Review of attachment 688027 [details] [diff] [review]:
-----------------------------------------------------------------

Heh, I was just about to submit a patch to you for review :p

r+ with comments addressed below

::: mozharness/mozilla/testing/errors.py
@@ +47,5 @@
>      {'regex': re.compile('''(TEST-INFO|TEST-KNOWN-FAIL|TEST-PASS|INFO \| )'''), 'level': INFO},
>  ]
> +
> +LogcatErrorList = [
> +    {'substr': 'Fatal signal 11 (SIGSEGV)', 'level': ERROR, 'explanation': 'This usually indicates the B2G process has crashed'},

{'substr': 'Fatal signal 7 (SIGBUS)', 'level': ERROR, 'explanation': 'This usually indicates the B2G process has crashed'},
{'substr': '[Javascript Error:', 'level': WARNING}

The two errors above should be added. Also they should probably be removed from the error_list defined in the two mozharness scripts as well.

Comment 12

[Andrew Halberstadt [:ahal]]

I also made a patch which starts to pull commonalities (including this logcat stuff) from the marionette and unittest scripts into a B2GMixin. I'll attach that to a separate bug once I've had a chance to test it

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 688027 [details] [diff] [review]
run logcat through run_command() to parse for errors

http://hg.mozilla.org/build/mozharness/rev/2e295e1add1f

Should fix this bug.

Comment 14

[Aki Sasaki (not active)]

(In reply to Andrew Halberstadt [:ahal] from comment #12)
> I also made a patch which starts to pull commonalities (including this
> logcat stuff) from the marionette and unittest scripts into a B2GMixin. I'll
> attach that to a separate bug once I've had a chance to test it

Awesome, I think that would be a good thing to do.

Comment 15

[Andrew Halberstadt [:ahal]]

This looks fixed: https://tbpl.mozilla.org/php/getParsedLog.php?id=17599768&tree=Mozilla-Inbound

Thanks!