530055 - Enable crash stack reporting on mozmill tests in tinderbox

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect)

Description

[Mark Banner (:standard8)]

Crash stack reporting won't currently work with mozmill. To get it to work we probably want to do a combination of the following:

1) Fix the unittest mozconfigs for Mac to specify -gdwarf-2 (unit test build boxes already do make buildsymbols).
2) Switch the runtest.py for mozmill from automation.process to automation.runApp.

I've not looked at item 2 in detail, if we can't do that we may have to do a combination of copy/pasting and setting environment variables, but hopefully not.

Comment 1

[Mark Banner (:standard8)]

Actually, comment 0 seems completely bogus.

The main part to do here is to implement a call to the checkForCrashes function as we do in the bloat tests:

http://mxr.mozilla.org/comm-central/source/mailnews/test/performance/bloat/runtest.py#222

I think that should get all the information necessary.

iirc this can be tested to some extent by creating a file in the PROFILE/minidumps directory (it just won't actually be able to process the stack).

Comment 2

[Rain [:rain1] (inactive)]

One thing I noticed while writing this patch was that automationutils specifies that --symbols-path should be absolute <http://mxr.mozilla.org/comm-central/source/mozilla/build/automationutils.py#82>, but we seem to pass in a relative path to checkForCrashes for the bloat tests. This is because DIST is relative at <http://mxr.mozilla.org/comm-central/source/mailnews/testsuite-targets.mk#44> -- and this is easily verified by running make mailbloat or by looking at tinderbox output.

Should this be fixed?

Comment 3

[Mark Banner (:standard8)]

(In reply to comment #2)
> Should this be fixed?

It probably doesn't hurt to fix it, but I do know (from a bustage a couple of days ago) that bloat crash stacks are working on trunk.

Comment 4

[Rain [:rain1] (inactive)]

Attachment: patch, v1

Tested with a fake dump inside the minidumps folder.

Comment 5

[Mark Banner (:standard8)]

Comment on attachment 428729 [details] [diff] [review]
patch, v1

>+        if checkForCrashes(os.path.join(self._profile_dir, 'minidumps'),
>+                           SYMBOLS_PATH, TEST_NAME):
>+            print >> sys.stderr, 'TinderboxPrint: ' + TEST_NAME + '<br/><em class="testfail">CRASH</em>'
>+            sys.exit(-1)

nit: -1 isn't supported by windows, 1 would be better.

Comment 6

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/comm-central/rev/3ce9d0c923d4

Comment 7

[Rain [:rain1] (inactive)]

Attachment: catch exceptions raised by mozmill too

What seems to be happening with failures like <http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1267204120.1267207018.18085.gz#err101> is that MozMill is raising an exception much before it decides to clean up. This patch should make sure the crash detection code is executed regardless of whether MozMill throws an exception.

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 429170 [details] [diff] [review]
catch exceptions raised by mozmill too

Yep, let's give this a try.

Comment 9

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/comm-central/rev/4b3a1e2c92eb

It appears the only issue remaining is to get runtestlist.py to actually pass symbols path to runtest.py....

Comment 10

[Rain [:rain1] (inactive)]

Attachment: pass in the symbols path from runtestlist.py

ouch, how in the world did I miss that :(

Comment 11

[Mark Banner (:standard8)]

Comment on attachment 429297 [details] [diff] [review]
pass in the symbols path from runtestlist.py

This looks better. Checked in as well:

http://hg.mozilla.org/comm-central/rev/400c636c00c9

Comment 12

[Rain [:rain1] (inactive)]

Attachment: pass in absolute paths

This patch is on top of the one in bug 545085.

(I actually tested this locally :p )

Comment 13

[Rain [:rain1] (inactive)]

Landed: http://hg.mozilla.org/comm-central/rev/177f77dc82c1

Comment 14

[Rain [:rain1] (inactive)]

Confirmed to work: http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTest/1267855861.1267858631.14684.gz