1214301 - Investigate how to take advantage of the minidump_stackwalk program to help SeaMonkey testing and QA on client-side

Status: NEW

(SeaMonkey :: Project Organization, enhancement)

Description

[Tony Mechelynck [:tonymec]]

Some builds of Firefox, Thunderbird and SeaMonkey are built "without symbols", meaning that their symbols cannot be resolved server-side by Socorro when a crash is reported. In most cases, however, these builds have a crashreporter-symbols.zip file by means of which it is possible to painstakingly resolve symbols by hand when reporting a crash to Bugzilla — if the crashreporter-symbols.zip hasn't been lost (for instance by installing a different hourly or own-compiled build) between crashing and reporting.

IMHO it would be a valuable enhancement to Breakpad if it could somehow (1) be told where the crashreporter-symbols.zip can be found, and (2) automatically resolve the symbols by means of it when reporting a crash.

Comment 1

[Tony Mechelynck [:tonymec]]

Somehow Bugzilla turned all/all to all/Android which is not what I meant.

Comment 2

[Tony Mechelynck [:tonymec]]

IRC exchange on #seamonkey (times in CEST, UTC+0200):
[Tue 2015-10-13 20:33:01] KaiRo   tonymec: I'm for WONTFIXing that one
[Tue 2015-10-13 20:33:13] tonymec too bad
[Tue 2015-10-13 20:33:36] KaiRo   but I won't do anything on that bug, it's out of what I can do anyhow
[Tue 2015-10-13 20:33:58] tonymec KaiRo: well, maybe someone else can?
[Tue 2015-10-13 20:34:07] KaiRo   the proper fix for your case is getting SeaMonkey Nightly builds back
[Tue 2015-10-13 20:34:31] KaiRo   tonymec: I hope nobody we pay will spend time on this as we have a lot of other things I consider higher priority that need to be done
[Tue 2015-10-13 20:35:00] KaiRo   volunteers can do whatever they like to, of course
[Tue 2015-10-13 20:37:09] tonymec KaiRo: getting back nightlies would be great, and I'm sure ewong is working on that, but of course when DLing an hourly one can't know in advance that it won't crash. I intentionally reported that problem with "enh" severity, and if some volunteer could do it I would heartily applaude.

Comment 3

[(not currently active) Ted Mielczarek]

I'm not really sure what part of things you mean when you say "Breakpad" here. If you're asking for the crash reporter to be able to resolve these symbols client-side before it submits the report to crash-stats.mozilla.com, then this would be WONTFIX--that's not a scenario that we're interested in supporting.

If you just want a way to get a stack out given a minidump + a symbols.zip, then we already have that in mozcrash, although it's not intended to be a user-facing feature. We use it in our test harnesses:
http://mozbase.readthedocs.org/en/latest/mozcrash.html#mozcrash.check_for_crashes

You pass it a directory that might contain minidumps, a symbols path (which can be a URL to a symbols.zip), and a path to a minidump_stackwalk binary (which you can build from the Breakpad source, or we have binaries in tooltool), and it will process each dump it finds locally with those symbols and dump the stack to stdout.

Comment 4

[Tony Mechelynck [:tonymec]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> I'm not really sure what part of things you mean when you say "Breakpad"
> here. […]

I usually say "Breakpad" for the client-side half of the crash reporter and "Socorro" for the server-side part; however from what you're saying in your comment other fixes would suit me than what I had originally thought of, so let me explain in more detail:

I upgrade my Mozilla apps by means of the following procedure (outlined here for SeaMonkey, but it applies /mutatis mutandis/ to Firefox and Thunderbird too):

1. Browse to http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk/ and see if there is a newer build than the one I'm currently using. If there is, go to step 4.
2. If not, browse to ../../tinderbox-builds/comm-central-trunk-linux64/ and find the newest subdirectory containing a full build and not only a build log.
3. If none, nothing to do.
4. Download (for the latest branch) the .tar.bz2, the .txt, the .json, the .mozinfo.json, and, if there is one, the crashreporter-symbols.zip
5. If my existing build of that app is currently running (for instance, if I used it to download the new build), hit Ctrl+Q
6. Watch gnome-system-monitor until there is a sharp drop in memory use.
7. If "ps -lC seamonkey" (or the equivalent for other apps) has more to say than just the column headings, go back to step 6.
8. rm -Rv /usr/local/seamonkey && tar -jxvf seamonkey-2.41a1.en-US.linux-x86_64.tar.bz2 -C /usr/local || echo 'exit status' $? ; date
9. Start (the new version of) the app.

When a crash happens, the checkbox "Tell Mozilla about this crash" is left ticked in the Breakpad dialog.

For a nightly from Mozilla, normally it's no sweat: Socorro has the symbols.

For a tinderbox-build, or for one of A. Kalla's "nightlies", Socorro doesn't have the symbols, but there is a crashreporter-symbols.zip. By opening that in Vim (which temporarily unzips the desired content) I can extract the symbols manually in order to write a "usable" bug report, it isn't hard but it is tedious.

Anything which could ease the procedure described in the above paragraph would be useful to me, and, I suppose, for any QA tester desiring to make his/her crash bug reports as useful as possible, even when Socorro doesn't have the desired symbols. I had thought of telling Breakpad where I have downloaded the crashreporter-symbols.zip so that (with an appropriate enhancement) it could report symbols instead of bare binary addresses. Not sure how involved this would be.

But anything which could automatize the procedure of generating a stack dump "with symbols" from a crash GUID and a crashreporter-symbols.zip would enormously ease my tester's job. Even if the whole procedure was carried on my own computer, to produce some text I could add as a Bugzilla comment or attachment.

I'm not sure what you call a minidump, or where it can be found. Maybe the pairs of *.dmp and *.extra binary files in <profile>/../Crash Reports/pending/ ? Pairing that (if it is what you call a minidump) and the corresponding crashreporter-symbols.zip as arguments to a minidump_stackwalk executable (is there one for linux-x86_64, or at least for linux-i686, that I can download?) would make my tester's job much easier. It would be an acceptable solution to the problem for which I wrote this RFE. After testing the procedure, if I think that a reasonably intelligent SeaMonkey user can be expected to follow it without error, I might even add a section describing it to the page which is temporarily at https://wiki.mozilla.org/User:Tonymec/Triage_HowTo so that other testers could profit by it.

Comment 5

[Tony Mechelynck [:tonymec]]

In bug 1218205 comment #0 I added the symbols by hand, Socorro apparently hadn't got them, or maybe it had already lost them. Happily the stack wasn't too deep, and the expertise I got some 30 or 40 years ago with COBOL link listings and binary crash addresses still serves me somewhat.

But the longer the stack, the more tedious the work. You say it can be automated. Where do I find the necessary software and how do I use it?

Comment 6

[Jesse Ruderman]

Point
https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py

at e.g.
https://dxr.mozilla.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/releng.manifest

and then point the resulting minidump_stackwalk binary at your minidump and symbols the same way mozcrash.py does (or call the functions in mozcrash.py).

Comment 7

[Tony Mechelynck [:tonymec]]

Thanks a lot Jesse.

After some guidance by you on IRC I could indeed donwload an ELF binary which tooltool.py renamed linux64-minidump_stackwalk (with one dash and one underscore!) after verifying its integrity. The next time I get a "crash without symbols" I'll reread the readthedocs article mentioned in comment 3, see if I can make sense of it and possibly find more info, probably at wikimoz, and finally see if I can generate a human-readable stackdump for Bugzilla with less hassle than I used to do by searching the temporarily-unzipped contents of a crashreporter-symbols.zip for the symbol with the highest value not exceeding each address reported by Breakpad.

Ted said in comment #3 that "this is not intended to be a user-facing feature". I shall assume then that my fellow testers and QA volunteers in the Firefox and Thunderbird teams will, alas for them, not have easy access to it.

In my official capacity as QA peer for SeaMonkey, my opinion is that, if this minidump_stackwalk program can make sense of a crashreporter-symbols.zip file and of whatever is stored locally at the time of a crash, then it would be a very valuable help, at least for technical-minded testers such as me and my fellow testers, who may happen to get a crash even on a tinderbox-build or on an own-compiled build of (Firefox, Thunderbird or) SeaMonkey, and would like to get a usable stack dump out of it rather that a list of hex offsets relative to libxul.so with here and there maybe a hex reference to the "seamonkey" binary or to some system library.

If I come around to test this program, and if the result is positive, I may propose it to the SeaMonkey Council as an addition for an upcoming overhaul to the now badly outdated "SeaMonkey Debug & QA UI" extension shipped as an integral part of trunk, and I think aurora, builds of SeaMonkey (as, on my system, /usr/local/seamonkey/distribution/extensions/[e-mail]). It may take some time though, because the crashing bugs I'm encountering at the moment (and there are some) are only intermittent and (on my system) low-frequency ones. (I am hitting annoying solid bugs but they aren't crashing bugs.)

Comment 8

[Tony Mechelynck [:tonymec]]

Attachment: "Interesting" URLs found so far

for future reference, by me or by whoever will be participating in this project

Comment 9

[Tony Mechelynck [:tonymec]]

Comment on attachment 8688403 [details]
"Interesting" URLs found so far

- Most of these files are documentation, of varying relevance. I haven't found anything both clear and to the point so far. Some of the docs are very Windows-minded; others mention Linux and Mac but usually only "in passing".
- The stackwalk.cc source can also be used as documentation by anyone with a passive knowledge of C++; but there are no comments in it.
- To get the minidump_stackwalk binary, run "tooltool.py -m <manifest> fetch" where <manifest> is the file "releng.manifest" found in the appropriate subdir of tooltool-manifests/
- The breakpad/ directory in the first hg.m.o URL quoted contains binaries. Not useful except probably after cloning.