Last Comment Bug 447771 - allow Linux distributions to submit crash reports to crash-stats.mozilla.com
: allow Linux distributions to submit crash reports to crash-stats.mozilla.com
Status: RESOLVED FIXED
[notacrash][crashkill-metrics]
:
Product: Core Graveyard
Classification: Graveyard
Component: Tracking (show other bugs)
: Trunk
: x86 Linux
: -- normal with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: chris hofmann
Mentors:
: 698288 (view as bug list)
Depends on: 437487 530041 535947
Blocks: 510358
  Show dependency treegraph
 
Reported: 2008-07-24 05:52 PDT by Ted Mielczarek [:ted.mielczarek]
Modified: 2016-07-15 12:13 PDT (History)
24 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Ted Mielczarek [:ted.mielczarek] 2008-07-24 05:52:57 PDT
Currently all Linux distributions disable our crash reporting. It's widely held that there are a lot more users running their distribution's version of Firefox than there are running the official Linux Firefox builds. I've heard interest expressed from asac (from Ubuntu) and Wolfgang (from openSUSE) in submitting crash reports to our Socorro instance. I think this would be a win-win situation, as we're not currently getting visibility into the crashes our Linux users are actually hitting, and the distros (afaik) don't have the kind of aggregate reporting we do.

This is filed as a tracking bug, because there are going to be a few things necessary on both sides to make this happen, and each distro that wants to participate will have to do a few of the same things.

Brief list of what we'll need:
1) Need a "distributor" field in the Socorro database, so we can distinguish official mozilla.org builds from distro-produced builds (and distinguish distro builds from each other) Probably also want a distro_version, for their use. (bug 437487)
2) Distros will need access to upload symbols for their builds and the system libraries they depend on to our symbol store. This shouldn't be that hard, but I don't know what the space requirements are going to look like, we'll need to coordinate with IT on that. Additionally, distros will have to sort out automated upload of symbols for their packages, since our Breakpad setup doesn't have on-demand symbol supplying.
3) Clearly we'll need to sort out the load situation on our Socorro processor, since we are currently extremely backlogged, and throwing more reports at it isn't a great idea until we have some server-side throttling in place.
Comment 1 Alexander Sack 2009-08-14 08:26:52 PDT
(In reply to comment #0)
> Currently all Linux distributions disable our crash reporting. It's widely held
> that there are a lot more users running their distribution's version of Firefox
> than there are running the official Linux Firefox builds. I've heard interest
> expressed from asac (from Ubuntu) and Wolfgang (from openSUSE) in submitting
> crash reports to our Socorro instance. I think this would be a win-win
> situation, as we're not currently getting visibility into the crashes our Linux
> users are actually hitting, and the distros (afaik) don't have the kind of
> aggregate reporting we do.
> 
> This is filed as a tracking bug, because there are going to be a few things
> necessary on both sides to make this happen, and each distro that wants to
> participate will have to do a few of the same things.

> 2) Distros will need access to upload symbols for their builds and the system
> libraries they depend on to our symbol store. This shouldn't be that hard, but
> I don't know what the space requirements are going to look like, we'll need to
> coordinate with IT on that. Additionally, distros will have to sort out
> automated upload of symbols for their packages, since our Breakpad setup
> doesn't have on-demand symbol supplying.

Is there any way we need to flag those symbols so you can match the right versions? I guess they should be somehow associated with a build/distro/distro-version id? For instance, our symbols for hardy might be different than those for jaunty. Same for 3.5.2 vs. 3.5.0.

Let's understand that first and I will see how we can upload them automatically.

> 3) Clearly we'll need to sort out the load situation on our Socorro processor,
> since we are currently extremely backlogged, and throwing more reports at it
> isn't a great idea until we have some server-side throttling in place.

You think that ubuntu/distro user base is high enough to make a big difference?
Comment 2 Benjamin Smedberg [:bsmedberg] 2009-08-14 08:34:10 PDT
> Is there any way we need to flag those symbols so you can match the right
> versions? I guess they should be somehow associated with a
> build/distro/distro-version id? For instance, our symbols for hardy might be
> different than those for jaunty. Same for 3.5.2 vs. 3.5.0.

In terms of finding the right symbols, no. The crash report contains a shared library signature which should be different.

But we'd probably want to have a manifest file of which symbol files belong to which releases, just for maintenance and eventual cleanujp.
Comment 3 Ted Mielczarek [:ted.mielczarek] 2009-08-14 08:49:43 PDT
As bsmedberg said, symbols on Linux are identified by an md5sum of some bits of the ELF file, so there's no need for any sort of mapping. The symbolstore.py script that we use to generate our symbols for nightly/release builds does produce a manifest file that we use for cleanup:
http://symbols.mozilla.org/firefox/firefox-3.5-WINNT-20090624025744-symbols.txt
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py

(In reply to comment #1)
> You think that ubuntu/distro user base is high enough to make a big difference?

Not now, but when I wrote that we were having serious issues keeping up with our existing load, so any additional user base might have been a problem. I think we've got that fairly well under control, and once my patch to resubmit reports (bug 378528) lands, we'll be able to throttle client-side and be in even better shape.

In short, I think we're in a good place to make this happen now. Let's figure out how to do it!
Comment 4 Jan Horak 2009-09-18 05:30:34 PDT
Fedora has recently developed its own system-wide automatic bug reporting tool - Abrt (https://fedorahosted.org/abrt/wiki). It can be used to automatically post crash reports to http://crash-stats.mozilla.com by writing a plugin. To create bug reports it use/download debug info packages for correspondent package for Fedora to user computer. If it is preferred approach for Fedora distribution I can try to do the plugin for http://crash-stats.mozilla.com. The benefit for  Fedora users it allows to do bug reports from any application by similar way.
Comment 5 Ted Mielczarek [:ted.mielczarek] 2009-09-18 05:34:55 PDT
Our crash reporting system works the opposite way, sort of. We simply generate a minidump on the client side that includes the raw stack data and registers, and upload it to our crash report server, which has debug symbols that are uploaded from when we build the binaries.

All we really need to get useful crash reports from your users is:
1) You run the "make buildsymbols" in the Firefox (or xulrunner) objdir before packaging the build, and upload the symbols to our symbol server (we have "make uploadsymbols" for that, as well)
2) You start building without --disable-crashreporter, which will let your builds use our existing crash reporting mechanism to submit to our crash reporting server.
Comment 6 Ted Mielczarek [:ted.mielczarek] 2009-09-18 05:36:58 PDT
And, to answer your more direct question, no, our crash reporting server is not setup to accept stack traces produced in the manner you describe. It only accepts binary minidumps currently.
Comment 7 Jan Horak 2009-09-18 06:16:55 PDT
I see. I'm trying to bring up this issue in Fedora Devel list if it is possible or how can we work on it. Thanks for comments.

https://www.redhat.com/archives/fedora-devel-list/2009-September/msg00712.html
Comment 8 Ted Mielczarek [:ted.mielczarek] 2009-09-18 07:20:01 PDT
Feel free to CC me on any discussions.
Comment 9 Jan Horak 2009-10-12 05:40:17 PDT
I tried to do the 'make buildsymbols' with --enable-crashreporter. I'm unable to generate debug info unless I add -gstabs+ or -gstabs to CFLAGS. Where should I add this gcc parameters and which one should I choose (for Firefox and Thunderbird)?
Comment 10 Ted Mielczarek [:ted.mielczarek] 2009-10-12 06:29:31 PDT
Right, you'll need to build with debug symbols, which requires stabs symbols currently (we have DWARF support coming soon). If you're building with a mozconfig, you can pass that in CFLAGS/CXXFLAGS in your mozconfig, otherwise if you're just running configure manually, you can do:
CFLAGS=-gstabs+ CXXFLAGS=-gstabs+ ../configure ...
Comment 11 Jan Horak 2009-10-21 06:14:16 PDT
We can put thunderbird-*crashreporter-symbols.zip into our debuginfo package (for example thunderbird-debuginfo-3.0-3.10.b4.fc12.x86_64.rpm) which is located at http://kojipkgs.fedoraproject.org/packages/thunderbird/3.0/ (according to release and architecture). Are you able to pull these files and retrieve your symbols  from this package (or would you like some assistance)?
Comment 12 Ted Mielczarek [:ted.mielczarek] 2009-10-21 06:15:56 PDT
It would be a lot easier if we just gave you an account and you uploaded the symbols to us instead.
Comment 13 Jan Horak 2009-10-21 07:16:07 PDT
Okay, but doing it manually is a bit unreliable. Looking at upload_symbols.sh, the script use ssh to upload files and it requires stored private key (so we can't make it public). I will have to discuss it with Fedora release engineering and then let you know.
Comment 14 Ted Mielczarek [:ted.mielczarek] 2009-10-21 07:28:36 PDT
Well, it just requires a private key be available on the box that does the uploading. You can see our production configs here:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/config.py#234

It just points to the private key which is installed on our build slaves.
Comment 15 Ted Mielczarek [:ted.mielczarek] 2009-10-21 07:29:30 PDT
Also, I didn't intend for it to be done manually, I was thinking you'd do this as an automated step as part of your build process, just like we do. Trying to set something up where we have to download your symbols and then install them after you've uploaded your build seems a lot more fragile.
Comment 16 Jesse Ruderman 2009-11-16 00:26:37 PST
See also:
* Bug 511713, "When crash reporter is disabled, about:crashes should say that"
* https://bugs.launchpad.net/ubuntu/+source/firefox-3.0/+bug/246843
Comment 17 Matěj Cepl 2009-12-16 01:33:49 PST
(In reply to comment #15)
> Also, I didn't intend for it to be done manually, I was thinking you'd do this
> as an automated step as part of your build process, just like we do. Trying to
> set something up where we have to download your symbols and then install them
> after you've uploaded your build seems a lot more fragile.

We would have to do some changes in our building process (or maybe somewhere later) ... building process is not allowed to access network (for reasons which shouldn't be that difficult to understand, I hope). That's not insurmountable obstacle, just to note what needs to be done.
Comment 18 Ted Mielczarek [:ted.mielczarek] 2009-12-16 04:33:13 PST
Ok. So, presumably, they have access to some server to which they upload the build results, right? If you can simply call "make buildsymbols" in the object directory, you'll wind up with a file in dist/ with a similar name to the build package, but ending in .crashreporter-symbols.zip. (You can see some of them in the upload directories here: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/ )

If you upload that symbol package along with the build, you could have a post-build process run elsewhere that uploads the contents of that zip to our symbol server. Our symbol upload script is quite simple:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.sh

We just run that script, passing the crashreporter-symbols.zip as the only argument, with some environment variables (described in the script) indicating the symbol server hostname and user login etc. You could use that or roll your own, it doesn't really matter.
Comment 19 Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny] 2010-04-13 12:46:56 PDT
*** Bug 510358 has been marked as a duplicate of this bug. ***
Comment 20 Jan Horak 2010-04-28 06:26:23 PDT
While we can build Firefox 3.6.4 on x86_64 now we would like to upload required debug info into your servers. It would be nice to get special account for Fedora. We are going to dedicate a machine to do this task automatically driven by our release system. Access to this machine will be restricted. Is it possible to create such account?
Comment 21 Ted Mielczarek [:ted.mielczarek] 2010-04-28 07:35:51 PDT
Sure, please file a bug just like bug 535947. You'll need to attach an ssh public key. Please CC me on the bug.
Comment 22 Wayne Mery (:wsmwk, NI for questions) 2011-11-28 08:10:17 PST
*** Bug 698288 has been marked as a duplicate of this bug. ***
Comment 23 russianneuromancer 2012-01-07 19:44:14 PST
Looks like Firefox 9.0 from Ubuntu repository already can submit crash reports but for Firefox-trunk from Mozilla daily PPA this feature is still disabled. Is there any reason for that?
Comment 24 Ted Mielczarek [:ted.mielczarek] 2012-01-07 19:45:59 PST
I don't know, you'd have to ask the Ubuntu Firefox maintainers.

Note You need to log in before you can comment on or make changes to this bug.