510358 - Accept crashes from Fedora distro Thunderbird builds

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Wayne Mery (:wsmwk)]

Determine and implement what is needed to get crash reports from Fedora builds (and perhaps other *major* linux distros)

Discussed briefly via email ... 

Thunderbird is very much interested in getting crash reports from builds of *major* linux distros.  

Our testing community is relatively small, so seeing linux distro crashes would greatly improve our ability to encourage and support linux testers (nightlies and betas), and improve the top crash list such that we can better target the right crashes for attention.  We also want a more accurate picture of Thunderbird's stability with average customers using released builds from those distros. But I think the testing community is our initial target.  Chris Aillon is also interested in making this happen for Fedora.  User stats: https://fedoraproject.org/wiki/Statistics

Perhaps this can be done within the existing Socorro environment, similar to what is being done with Eudora/Penelope.

Comment 1

[Philippe M. Chiasson (:gozer)]

(In reply to comment #0)
> Determine and implement what is needed to get crash reports from Fedora builds
> (and perhaps other *major* linux distros)
> [...]
> 
> Perhaps this can be done within the existing Socorro environment, similar to
> what is being done with Eudora/Penelope.

In the case of Eudora, they generate the buildsymbols and upload them to the MoCo symbol server directly. I am not sure if that's a possiblilty in this case, but I believe there is a simpler/better solution.

If as part of the distro building the binary packages, the buildsymbols are created and included in the -devel/-debug packages. Then *we* could pull these binary packages, crack them open, extract the buildsymbols from them and upload them to the symbol server ourselves.

The distro builds would also need to have crashreporting enabled.

Comment 2

[Frank Griswold [:griswolf] [:fgriswold]]

Aside from symbol information, the socorro crash handling code is already suitable for handling these crashes, assuming (as of right now):
 - the product and version strings are unchanged in distro builds
 - the os_name is associated with the product/version in our productdims table

Soon, we will have a modified database schema which does not conflate os_name with product information. It will handle these crashes assuming:
  - the product and version strings are unchanged in distro builds

Now or later, if product or version is changed, we will still be able to handle crashes by adding the changed strings to our config tables.

Comment 3

[Samuel Sidler (old account; do not CC)]

While the product and version strings are the likely same, we probably want to call these something separate since (last I heard) Linux distros modify Thunderbird (and Firefox for that matter) before shipping it. They'll have their own symbols not for general "Linux" (which we already have), but for the binaries they actually ship. I don't think we can easily differentiate, but I'll like a Socorro person weigh in.

I'm not sure this is a Webtools::Socorro bug anyway...

Comment 4

[(not currently active) Ted Mielczarek]

Might be a dupe of bug 447771. We want this for Firefox as well. We do have  "distributor" and "distributor_version" fields in the db. My thought was that we would accept/store/display those, so that we could distinguish distro builds from stock Mozilla builds.

Comment 5

[Frank Griswold [:griswolf] [:fgriswold]]

current time: The product has an os_name ('Linux') component. It is very difficult to distinguish aggregated crash data based on os_version.

near future: The product is a name and version in the productdims table but there is also an osdims table that knows both os_name and os_version

Based on discussion with (ss?) the Linux version is reduced from the raw version info to something like "2.6.27.25 i686" or "2.6.27.23 x86_64". Queries on materialized views can be constructed that distinguish data by os_name and os_version.

Example version string simplifications:
Raw: 0.0.0 Linux 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 00:28:35 UTC 2009 i686 GNU/Linux
becomes: 2.6.28 i686

Raw: Linux 0.0.0 Linux 2.6.18-1.2798.fc6xen #1 SMP Mon Oct 16 15:11:19 EDT 2006 i686 GNU/Linux
becomes: 2.6.18 i686

at present, distibutor and distributor_version are present but unused.

Comment 6

[Christopher Aillon (sabbatical, not receiving bugmail)]

Jan Horak will be working on this for the Fedora side of things, and helping out where needed.

Comment 8

[Wayne Mery (:wsmwk)]

bug  447771 is blocked by this. reopening

Comment 9

[Ludovic Hirlimann [:Usul]]

Well to my knowledge Suse does this - so can we get it also for Fedora ?

Comment 10

[Christopher Aillon (sabbatical, not receiving bugmail)]

So, we wanted to turn this on for both 32 and 64 bit, but using Moz 1.9.x which uses the gstabs format, we had run into some problems with producing 64 bit rpms.

https://bugzilla.redhat.com/643305
https://bugzilla.redhat.com/453506 

While 643305 was able to get resolved, 453506 got stalled, and so we've modified our build to enable the crash reporter and -gstabs+ only under x86-32.

For Moz 2.0, which uses the gdwarf format, we're able to send symbols no problem.  We're already sending FF4 data for both arches, and when we move to a Moz 2.0 based TB, I don't expect any problems.

I think this bug can be closed now.

Comment 11

[(not currently active) Ted Mielczarek]

1.9.2 should have all the patches necessary to use -gdwarf-2. They got backported as part of the "out of process plugins" effort.

Comment 12

[Jan Horak [:jhorak]]

So there is no problem using gdwarf even when official builds use gstabs [1]?

[1] http://hg.mozilla.org/build/buildbot-configs/file/fc842735184a/mozilla2/linux/mozilla-1.9.2/release/mozconfig

Comment 13

[(not currently active) Ted Mielczarek]

Yeah, I think we just didn't switch those configs. You'll note that the 1.9.2 64-bit nightly config is a symlink to the mozilla-central nightly config:
http://hg.mozilla.org/build/buildbot-configs/file/25a52018d139/mozilla2/linux64/mozilla-1.9.2/nightly
http://hg.mozilla.org/build/buildbot-configs/file/25a52018d139/mozilla2/linux64/mozilla-central/nightly/mozconfig

which uses DWARF. (We didn't ship a 64-bit Linux release off of 1.9.2, so there's no release config to show.)