1171427 - Update minidump_stackwalk binaries in tooltool + tooltool manifest in tree

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

The minidump_stackwalk binaries we're using in CI are a little bit out of date. RyanVM linked me a test log where we failed to get a stack out of a minidump:
https://treeherder.mozilla.org/logviewer.html#?job_id=1602425&repo=mozilla-central

The root cause there was that one of the threads in the minidump had bad stack memory listed:
00:11:08 INFO - 2015-06-04 00:11:08: minidump.cc:1357: ERROR: MinidumpThread has a memory region problem, 0x0+0x0 
00:11:08 INFO - 2015-06-04 00:11:08: minidump.cc:1551: ERROR: MinidumpThreadList cannot read thread 55/81 


I fixed this in upstream Breakpad a while back, making the stackwalker able to print stacks for other threads even if some threads can't be handled properly. I was able to get a stack out of this dump locally with a minidump_stackwalk built from a current breakpad checkout. We just need to update the minidump_stackwalk binaries we're using, which are in tooltool nowadays.

The tooltool manifests live here:
https://dxr.mozilla.org/mozilla-central/source/testing/config/tooltool-manifests

Comment 1

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5d31c00857aa

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: MozReview Request: bug 1171427 - update minidump_stackwalk binaries. r?jgriffin

bug 1171427 - update minidump_stackwalk binaries. r?jgriffin

Comment 3

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8631200 [details]
MozReview Request: bug 1171427 - update minidump_stackwalk binaries. r?jgriffin

https://reviewboard.mozilla.org/r/12857/#review11441

There are EOL extra spaces most places here, but otherwise looks good.

Comment 4

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=49a82ae758ce

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> Comment on attachment 8631200 [details]
> MozReview Request: bug 1171427 - update minidump_stackwalk binaries.
> r?jgriffin
> 
> https://reviewboard.mozilla.org/r/12857/#review11441
> 
> There are EOL extra spaces most places here, but otherwise looks good.

That's just how the tooltool.py script generates the manifest for some reason.

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=49a82ae758ce

This broke on Linux and Mac because:
a) I forgot to statically link the Linux binaries
b) I used the wrong filename for the Mac binary so mozharness couldn't find it

Worked on Windows, though. I rebuilt the Linux binaries and fixed the Mac filename and am going to push to try again to sanity check that.

Comment 7

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5915eaa414bd

Comment 8

[(not currently active) Ted Mielczarek]

url:        https://hg.mozilla.org/integration/mozilla-inbound/rev/50be949e3494424ce551859913811de0292963fb
changeset:  50be949e3494424ce551859913811de0292963fb
user:       Ted Mielczarek <[e-mail]>
date:       Wed Jul 08 15:25:58 2015 -0400
description:
bug 1171427 - update minidump_stackwalk binaries. r=jgriffin

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=5915eaa414bd

That worked fine on Linux/Mac.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Backed out for WinXP test timeouts.
https://treeherder.mozilla.org/logviewer.html#?job_id=11560305&repo=mozilla-inbound

https://hg.mozilla.org/integration/mozilla-inbound/rev/3a9b803a09a8

Comment 11

[(not currently active) Ted Mielczarek]

I missed the WinXP bustage in my try push, sorry about that. Apparently the version of mingw-w64 I used to build the binaries has a bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=755448

I'll see if I can rebuild with a fixed version.

Comment 12

[(not currently active) Ted Mielczarek]

Talking with jacek on IRC he pointed out that it wasn't that bug I was seeing, but rather that in the Breakpad source we have some code that does:
#ifdef _WIN32
#define strtok_r strtok_s
#endif

..so this is Breakpad shooting me in the foot. I have a patch for that, I just rebuilt minidump_stackwalk binaries with it and will check them on try.

Comment 13

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=13a7bd50c6e6

Comment 14

[(not currently active) Ted Mielczarek]

That looks good! Hooray!

Comment 15

[(not currently active) Ted Mielczarek]

url:        https://hg.mozilla.org/integration/mozilla-inbound/rev/d9b9c8130397969040eb24f22bcf535024cd7ea4
changeset:  d9b9c8130397969040eb24f22bcf535024cd7ea4
user:       Ted Mielczarek <[e-mail]>
date:       Fri Aug 14 07:09:31 2015 -0400
description:
bug 1171427 - update minidump_stackwalk binaries in tooltool manifests. r=jgriffin

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/d9b9c8130397