806651 - mozharness unittest minidump stackwalk not working

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

17:15:57  WARNING -  TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_fullscreen-api.html | Exited with code 1 during test run
17:15:57  WARNING -  This is a harness error.
17:15:57     INFO -  INFO | automation.py | Application ran for: 0:08:43.839605
17:15:57     INFO -  INFO | automation.py | Reading PID log: /var/folders/vs/s48lhbg13cx0j819ycft495000000w/T/tmpibmhxJpidlog
17:15:57     INFO -  Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-macosx64/1351548997/firefox-19.0a1.en-US.mac.crashreporter-symbols.zip
17:15:59     INFO -  PROCESS-CRASH | /tests/content/html/content/test/test_fullscreen-api.html | application crashed (minidump found)
17:15:59     INFO -  Crash dump filename: /var/folders/vs/s48lhbg13cx0j819ycft495000000w/T/tmp2GVgs5/minidumps/E4651AEA-78D1-438C-9FB9-A222016BAD4F.dmp
17:15:59     INFO -  MINIDUMP_STACKWALK not set, can't process dump.
17:16:00     INFO -  WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!
17:16:00     INFO -  INFO | runtests.py | Running tests: end.
17:16:00    ERROR - Return code: 1
17:16:00     INFO - TinderboxPrint: mochitest-plain1<br/><em class="testfail">T-FAIL</em>

https://tbpl.mozilla.org/php/getParsedLog.php?id=16567097&tree=Cedar

Comment 1

[Aki Sasaki (not active)]

Attachment: (wip) partial config patch

I can either check in 14mb of binaries into mozharness, or depend on build/tools again.

This patch adds build/tools as a repo, and the minidump paths to the unittest config files.  Still to do: re-add hg support to desktop_unittest.py (including windows hg exe) and use the minidump* config items in env when running the tests.

Comment 2

[Aki Sasaki (not active)]

Attachment: enable minidump on unittests

Need to test in staging.

Comment 3

[Aki Sasaki (not active)]

[16:47]	<edmorley>	aki|away: is there a bug filed on making mozharness output the PROCESS-CRASH lines as an error, not INFO ? (eg https://tbpl.mozilla.org/php/getParsedLog.php?id=16604974&tree=Cedar)

This should do it:

diff --git a/scripts/desktop_unittest.py b/scripts/desktop_unittest.py
--- a/scripts/desktop_unittest.py
+++ b/scripts/desktop_unittest.py
@@ -6,25 +6,26 @@
 # ***** END LICENSE BLOCK *****
 """desktop_unittest.py
 The goal of this is to extract desktop unittestng from buildbot's factory.py
 
 author: Jordan Lund
 """
 
 import os
+import re
 import sys
 import copy
 import shutil
 
 # load modules from parent dir
 sys.path.insert(1, os.path.dirname(sys.path[0]))
 
 from mozharness.base.errors import BaseErrorList
-from mozharness.base.log import INFO
+from mozharness.base.log import INFO, ERROR
 from mozharness.base.vcs.vcsbase import MercurialScript
 from mozharness.mozilla.testing.testbase import TestingMixin, testing_config_op
 from mozharness.mozilla.testing.unittest import DesktopUnittestOutputParser
 
 SUITE_CATEGORIES = ['mochitest', 'reftest', 'xpcshell']
 
 
 # DesktopUnittest {{{1
@@ -315,19 +316,23 @@ class DesktopUnittest(TestingMixin, Merc
                 cmd = abs_base_cmd
                 replace_dict = {
                     'abs_app_dir': abs_app_dir,
                 }
                 for arg in suites[suite]:
                     cmd.append(arg % replace_dict)
                 suite_name = suite_category + '-' + suite
                 tbpl_status, log_level = None, None
+                error_list = BaseErrorList + [{
+                    'regex': re.compile(r'''PROCESS-CRASH.*minidump found'''),
+                    'level': ERROR,
+                }]
                 parser = DesktopUnittestOutputParser(suite_category,
                                                      config=self.config,
-                                                     error_list=BaseErrorList,
+                                                     error_list=error_list,
                                                      log_obj=self.log_obj)
                 env = {}
                 if c.get('minidump_stackwalk_path'):
                     env['MINIDUMP_STACKWALK'] = c['minidump_stackwalk_path']
                 if c.get('minidump_save_path'):
                     env['MINIDUMP_SAVE_PATH'] = c['minidump_save_path']
                 env = self.query_env(partial_env=env, log_level=INFO)
                 return_code = self.run_command(cmd, cwd=dirs['abs_work_dir'],

Comment 4

[Ed Morley [:emorley]]

Thank you :-)

Comment 5

[Aki Sasaki (not active)]

Attachment: unittest minidump + improved error list

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 678166 [details] [diff] [review]
unittest minidump + improved error list

http://hg.mozilla.org/build/mozharness/rev/5c5cd39b1131

Comment 7

[Aki Sasaki (not active)]

14:23:17     INFO -  MINIDUMP_STACKWALK binary not found: C:\talos-slave\test\build/tools/breakpad/win32/minidump_stackwalk

Comment 8

[Aki Sasaki (not active)]

Attachment: fix minidump stackwalk path on windows

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 681266 [details] [diff] [review]
fix minidump stackwalk path on windows

http://hg.mozilla.org/build/mozharness/rev/230bbacd4d0e