1238305 - cppunittests do not look up breakpad symbols for logged stack traces

Status: RESOLVED FIXED

(Testing :: CPPUnitTest, defect)

Description

[(No longer employed by Mozilla) Aaron Klotz]

I'm seeing situations like this:
https://treeherder.mozilla.org/logviewer.html#?job_id=3036257&repo=mozilla-central#L5518

We've hit an assertion but the logged call stack is of limited utility (without a lot of tedious manual intervention) due to lack of symbolication. We have the breakpad symbols for the build, but MozDescribeCodeAddress uses debughlp which groks pdb. If pdbs aren't around, we're not going to get much out of it.

I'm not sure what the best way forward is. Does breakpad have some kind of facility to act as a MS symbol server? Or do we need to add breakpad .sym parsing to MozDescribeCodeAddress?

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #0)
> I'm not sure what the best way forward is. Does breakpad have some kind of
> facility to act as a MS symbol server? Or do we need to add breakpad .sym
> parsing to MozDescribeCodeAddress?

Comment 2

[(No longer employed by Mozilla) Aaron Klotz]

(This also makes debugging in mozharness test environments more difficult, so I'd suggest that a symbol server option is probably more preferable than just modifying MozDescribeCodeAddress).

Comment 3

[(not currently active) Ted Mielczarek]

Our other harnesses just post-process assertion stacks using https://dxr.mozilla.org/mozilla-central/source/tools/rb/fix_stack_using_bpsyms.py, like:
https://dxr.mozilla.org/mozilla-central/rev/6020a4cb41a77a09484c24a5875bb221714c0e6a/testing/mochitest/runtests.py#2525

We should just add that same code to the C++ unittest harness:
https://dxr.mozilla.org/mozilla-central/source/testing/runcppunittests.py

Comment 4

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Patch

Locally on mach and on try by looking at the stacks in the assertions on this try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=75bf5d41f88e

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8709133 [details] [diff] [review]
Patch

Review of attachment 8709133 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for fixing this!

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/c6617c4a27b4

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out in https://hg.mozilla.org/integration/fx-team/rev/aeacd77c05b5 for android cpp test failures:
https://treeherder.mozilla.org/logviewer.html#?job_id=6917684&repo=fx-team

Comment 8

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d293233ade5

Comment 9

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=49cf7894832c

Comment 10

[(No longer employed by Mozilla) Aaron Klotz]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c32452df00e86d86e65b5097ea253f488816067
Bug 1238305: Modify cppunittests to look up breakpad symbols for logged stack traces; r=ted

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I had to back this out for cpp test failures (this time on Windows debug) like https://public-artifacts.taskcluster.net/dsz6T2p6S-KvcVrDdNySaw/0/public/logs/live_backing.log

https://hg.mozilla.org/integration/mozilla-inbound/rev/01927ad414ea0925f976b191c2db41a0d0152ae9

Comment 12

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1f95245958d025da1f5d3008eedc32434eddb9a

Comment 13

[(No longer employed by Mozilla) Aaron Klotz]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f4109f88bd226f8eb9d4459608973a346c4bfc2

Comment 14

[(No longer employed by Mozilla) Aaron Klotz]

https://hg.mozilla.org/integration/mozilla-inbound/rev/63bc837753d653ab13635e432400c1b6d31d12ca
Bug 1238305: Modify cppunittests to look up breakpad symbols for logged stack traces; r=ted

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/63bc837753d6