Closed Bug 921042 Opened 11 years ago Closed 11 years ago

Flash and FennecAndroid symbols are not used since some time on 2013-09-24

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: kairo, Assigned: selenamarie)

Details

This hinders analysis of a lot of crashes atm.
Severity: normal → blocker
The first npswf32_11_8_800_168.dll@0x343e9d report affected seems to have been submitted at 2013-09-24T18:22:38+00:00 - so I think very slightly before is when this all started.
Also, https://crash-stats.mozilla.com/report/list?signature=F1398665248_____________________________ which is the symbolized version, was seen the last time at 2013-09-24T21:08:08+00:00

solarce said on IRC that RabbitMQ was rolled out on that day "between 11:30am and 3:30pm" (Pacific, UTC-7) and that fits the timeframe of the issues in this bug.
-> % svn diff
Index: modules/socorro/files/prod/etc-socorro/processor.ini
===================================================================
--- modules/socorro/files/prod/etc-socorro/processor.ini	(revision 75760)
+++ modules/socorro/files/prod/etc-socorro/processor.ini	(working copy)
@@ -347,7 +347,7 @@
     # name: processor_symbols_pathname_list
     # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces)
     # converter: socorro.processor.legacy_processor.create_symbol_path_str
-    #processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g"
+    processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g"

     # name: stackwalk_command_line
     # doc: the template for the command to invoke minidump_stackwalk
bburton@althalus [09:13:31] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing regression in processor config for socorro prod, bug 921042"
Sending        trunk/modules/socorro/files/prod/etc-socorro/processor.ini
Transmitting file data .
Committed revision 75771.
Assignee: nobody → bburton
Missing some symbols!

-> % svn diff
Index: modules/socorro/files/prod/etc-socorro/processor.ini
===================================================================
--- modules/socorro/files/prod/etc-socorro/processor.ini	(revision 75771)
+++ modules/socorro/files/prod/etc-socorro/processor.ini	(working copy)
@@ -347,7 +347,7 @@
     # name: processor_symbols_pathname_list
     # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces)
     # converter: socorro.processor.legacy_processor.create_symbol_path_str
-    processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g"
+    processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo

     # name: stackwalk_command_line
     # doc: the template for the command to invoke minidump_stackwalk
bburton@althalus [09:18:17] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing regression in processor config for socorro prod, bug 921042"
Sending        trunk/modules/socorro/files/prod/etc-socorro/processor.ini
Transmitting file data .
Committed revision 75772.
Ok, I guess now we will need to reprocess crashes.

Are the zte ones meant to be included?
Group: mozilla-corporation-confidential
(In reply to Laura Thomson :laura from comment #6)
> Ok, I guess now we will need to reprocess crashes.
> 
> Are the zte ones meant to be included?

Yes. Why shouldn't they?
Ok so here's what we need to reprocess: diff the config we have now (thanks solarce) vs the defaults, and reprocess anything that uses the symbols we just added back, that came in on 9.24 or later.
-> % svn diff
Index: modules/socorro/files/prod/etc-socorro/processor.ini
===================================================================
--- modules/socorro/files/prod/etc-socorro/processor.ini	(revision 75772)
+++ modules/socorro/files/prod/etc-socorro/processor.ini	(working copy)
@@ -347,7 +347,7 @@
     # name: processor_symbols_pathname_list
     # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces)
     # converter: socorro.processor.legacy_processor.create_symbol_path_str
-    processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo
+    processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo"

     # name: stackwalk_command_line
     # doc: the template for the command to invoke minidump_stackwalk
bburton@althalus [10:06:53] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing missing closing double quote in symbols_pathname, in processor config for socorro prod, bug 921042"
Sending        trunk/modules/socorro/files/prod/etc-socorro/processor.ini
Transmitting file data .
Committed revision 75778.
Assignee: bburton → sdeckelmann
From what I saw on IRC, it looks like the default set is this:
/mnt/socorro/symbols/symbols_ffx,
/mnt/socorro/symbols/symbols_sea,
/mnt/socorro/symbols/symbols_tbrd,
/mnt/socorro/symbols/symbols_sbrd,
/mnt/socorro/symbols/symbols_os

And we have been using this since the start of this issue.

So, what we have not been using in this time is this set:
/mnt/socorro/symbols/symbols_mob,
/mnt/socorro/symbols/symbols_penelope,
/mnt/socorro/symbols/symbols_camino,
/mnt/socorro/symbols/symbols_solaris,
/mnt/socorro/symbols/symbols_opensuse,
/mnt/socorro/symbols/symbols_ubuntu,
/mnt/socorro/symbols/symbols_fedora,
/mnt/socorro/symbols/symbols_adobe,
/mnt/socorro/symbols/symbols_b2g,
/mnt/socorro/symbols/symbols_geeksphone,
/mnt/socorro/symbols/symbols_tclpartner,
/mnt/socorro/symbols/symbols_zte,
/mnt/socorro/symbols/symbols_leo

So, we need to reprocess everything where those are used.


This means at least all in FennecAndroid, WebappRuntimeMobile, and B2G products; all Flash plugin reports (as all others are few, that might just mean all plugin reports); and all Linux reports, submitted since 2013-09-24T18:22:38+00:00
Group: mozilla-corporation-confidential
The problem arose because of a configuration setting that was incorrectly commented out during our reconfiguration which enabled RabbitMQ.

The setting fell back to using default directories we specify in our source tree, leaving out things like our ZTE, linux and various other important symbols.

The problem affected about ~120k crashes. We started reprocessing at about 12:38pm PT. Processing concluded at about 1:45pm PT, and I started the matview backfill shortly thereafter. 

9/25 is now complete. 9/24 has been started.
Backfill now complete.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Kairo can this be marked verified?
Flags: needinfo?(kairo)
Yes.
Status: RESOLVED → VERIFIED
Flags: needinfo?(kairo)
You need to log in before you can comment on or make changes to this bug.