Closed Bug 447771 Opened 16 years ago Closed 13 years ago

allow Linux distributions to submit crash reports to crash-stats.mozilla.com

Categories

(Core Graveyard :: Tracking, defect)

Product:
Core Graveyard
Component:
Tracking
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ted, Unassigned)

References

Details

(Whiteboard: [notacrash][crashkill-metrics]) 

Currently all Linux distributions disable our crash reporting. It's widely held that there are a lot more users running their distribution's version of Firefox than there are running the official Linux Firefox builds. I've heard interest expressed from asac (from Ubuntu) and Wolfgang (from openSUSE) in submitting crash reports to our Socorro instance. I think this would be a win-win situation, as we're not currently getting visibility into the crashes our Linux users are actually hitting, and the distros (afaik) don't have the kind of aggregate reporting we do.

This is filed as a tracking bug, because there are going to be a few things necessary on both sides to make this happen, and each distro that wants to participate will have to do a few of the same things.

Brief list of what we'll need:
1) Need a "distributor" field in the Socorro database, so we can distinguish official mozilla.org builds from distro-produced builds (and distinguish distro builds from each other) Probably also want a distro_version, for their use. (bug 437487)
2) Distros will need access to upload symbols for their builds and the system libraries they depend on to our symbol store. This shouldn't be that hard, but I don't know what the space requirements are going to look like, we'll need to coordinate with IT on that. Additionally, distros will have to sort out automated upload of symbols for their packages, since our Breakpad setup doesn't have on-demand symbol supplying.
3) Clearly we'll need to sort out the load situation on our Socorro processor, since we are currently extremely backlogged, and throwing more reports at it isn't a great idea until we have some server-side throttling in place.
(In reply to comment #0)
> Currently all Linux distributions disable our crash reporting. It's widely held
> that there are a lot more users running their distribution's version of Firefox
> than there are running the official Linux Firefox builds. I've heard interest
> expressed from asac (from Ubuntu) and Wolfgang (from openSUSE) in submitting
> crash reports to our Socorro instance. I think this would be a win-win
> situation, as we're not currently getting visibility into the crashes our Linux
> users are actually hitting, and the distros (afaik) don't have the kind of
> aggregate reporting we do.
> 
> This is filed as a tracking bug, because there are going to be a few things
> necessary on both sides to make this happen, and each distro that wants to
> participate will have to do a few of the same things.

> 2) Distros will need access to upload symbols for their builds and the system
> libraries they depend on to our symbol store. This shouldn't be that hard, but
> I don't know what the space requirements are going to look like, we'll need to
> coordinate with IT on that. Additionally, distros will have to sort out
> automated upload of symbols for their packages, since our Breakpad setup
> doesn't have on-demand symbol supplying.

Is there any way we need to flag those symbols so you can match the right versions? I guess they should be somehow associated with a build/distro/distro-version id? For instance, our symbols for hardy might be different than those for jaunty. Same for 3.5.2 vs. 3.5.0.

Let's understand that first and I will see how we can upload them automatically.

> 3) Clearly we'll need to sort out the load situation on our Socorro processor,
> since we are currently extremely backlogged, and throwing more reports at it
> isn't a great idea until we have some server-side throttling in place.

You think that ubuntu/distro user base is high enough to make a big difference?
> Is there any way we need to flag those symbols so you can match the right
> versions? I guess they should be somehow associated with a
> build/distro/distro-version id? For instance, our symbols for hardy might be
> different than those for jaunty. Same for 3.5.2 vs. 3.5.0.

In terms of finding the right symbols, no. The crash report contains a shared library signature which should be different.

But we'd probably want to have a manifest file of which symbol files belong to which releases, just for maintenance and eventual cleanujp.
As bsmedberg said, symbols on Linux are identified by an md5sum of some bits of the ELF file, so there's no need for any sort of mapping. The symbolstore.py script that we use to generate our symbols for nightly/release builds does produce a manifest file that we use for cleanup:
http://symbols.mozilla.org/firefox/firefox-3.5-WINNT-20090624025744-symbols.txt
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py

(In reply to comment #1)
> You think that ubuntu/distro user base is high enough to make a big difference?

Not now, but when I wrote that we were having serious issues keeping up with our existing load, so any additional user base might have been a problem. I think we've got that fairly well under control, and once my patch to resubmit reports (bug 378528) lands, we'll be able to throttle client-side and be in even better shape.

In short, I think we're in a good place to make this happen now. Let's figure out how to do it!
Fedora has recently developed its own system-wide automatic bug reporting tool - Abrt (https://fedorahosted.org/abrt/wiki). It can be used to automatically post crash reports to http://crash-stats.mozilla.com by writing a plugin. To create bug reports it use/download debug info packages for correspondent package for Fedora to user computer. If it is preferred approach for Fedora distribution I can try to do the plugin for http://crash-stats.mozilla.com. The benefit for  Fedora users it allows to do bug reports from any application by similar way.
Our crash reporting system works the opposite way, sort of. We simply generate a minidump on the client side that includes the raw stack data and registers, and upload it to our crash report server, which has debug symbols that are uploaded from when we build the binaries.

All we really need to get useful crash reports from your users is:
1) You run the "make buildsymbols" in the Firefox (or xulrunner) objdir before packaging the build, and upload the symbols to our symbol server (we have "make uploadsymbols" for that, as well)
2) You start building without --disable-crashreporter, which will let your builds use our existing crash reporting mechanism to submit to our crash reporting server.
And, to answer your more direct question, no, our crash reporting server is not setup to accept stack traces produced in the manner you describe. It only accepts binary minidumps currently.
I see. I'm trying to bring up this issue in Fedora Devel list if it is possible or how can we work on it. Thanks for comments.

https://www.redhat.com/archives/fedora-devel-list/2009-September/msg00712.html
Feel free to CC me on any discussions.
Blocks: 510358
I tried to do the 'make buildsymbols' with --enable-crashreporter. I'm unable to generate debug info unless I add -gstabs+ or -gstabs to CFLAGS. Where should I add this gcc parameters and which one should I choose (for Firefox and Thunderbird)?
Right, you'll need to build with debug symbols, which requires stabs symbols currently (we have DWARF support coming soon). If you're building with a mozconfig, you can pass that in CFLAGS/CXXFLAGS in your mozconfig, otherwise if you're just running configure manually, you can do:
CFLAGS=-gstabs+ CXXFLAGS=-gstabs+ ../configure ...
We can put thunderbird-*crashreporter-symbols.zip into our debuginfo package (for example thunderbird-debuginfo-3.0-3.10.b4.fc12.x86_64.rpm) which is located at http://kojipkgs.fedoraproject.org/packages/thunderbird/3.0/ (according to release and architecture). Are you able to pull these files and retrieve your symbols  from this package (or would you like some assistance)?
It would be a lot easier if we just gave you an account and you uploaded the symbols to us instead.
Okay, but doing it manually is a bit unreliable. Looking at upload_symbols.sh, the script use ssh to upload files and it requires stored private key (so we can't make it public). I will have to discuss it with Fedora release engineering and then let you know.
Well, it just requires a private key be available on the box that does the uploading. You can see our production configs here:
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/config.py#234

It just points to the private key which is installed on our build slaves.
Also, I didn't intend for it to be done manually, I was thinking you'd do this as an automated step as part of your build process, just like we do. Trying to set something up where we have to download your symbols and then install them after you've uploaded your build seems a lot more fragile.
Whiteboard: [notacrash][crashkill-metrics]
See also:
* Bug 511713, "When crash reporter is disabled, about:crashes should say that"
* https://bugs.launchpad.net/ubuntu/+source/firefox-3.0/+bug/246843
Depends on: 530041
(In reply to comment #15)
> Also, I didn't intend for it to be done manually, I was thinking you'd do this
> as an automated step as part of your build process, just like we do. Trying to
> set something up where we have to download your symbols and then install them
> after you've uploaded your build seems a lot more fragile.

We would have to do some changes in our building process (or maybe somewhere later) ... building process is not allowed to access network (for reasons which shouldn't be that difficult to understand, I hope). That's not insurmountable obstacle, just to note what needs to be done.
Ok. So, presumably, they have access to some server to which they upload the build results, right? If you can simply call "make buildsymbols" in the object directory, you'll wind up with a file in dist/ with a similar name to the build package, but ending in .crashreporter-symbols.zip. (You can see some of them in the upload directories here: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/ )

If you upload that symbol package along with the build, you could have a post-build process run elsewhere that uploads the contents of that zip to our symbol server. Our symbol upload script is quite simple:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.sh

We just run that script, passing the crashreporter-symbols.zip as the only argument, with some environment variables (described in the script) indicating the symbol server hostname and user login etc. You could use that or roll your own, it doesn't really matter.
Depends on: 535947
While we can build Firefox 3.6.4 on x86_64 now we would like to upload required debug info into your servers. It would be nice to get special account for Fedora. We are going to dedicate a machine to do this task automatically driven by our release system. Access to this machine will be restricted. Is it possible to create such account?
Sure, please file a bug just like bug 535947. You'll need to attach an ssh public key. Please CC me on the bug.
Looks like Firefox 9.0 from Ubuntu repository already can submit crash reports but for Firefox-trunk from Mozilla daily PPA this feature is still disabled. Is there any reason for that?
I don't know, you'd have to ask the Ubuntu Firefox maintainers.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.