Closed
Bug 921042
Opened 6 years ago
Closed 6 years ago
Flash and FennecAndroid symbols are not used since some time on 20130924
Categories
(Socorro :: General, task, blocker)
Socorro
General
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: kairo, Assigned: selenamarie)
Details
Some time late on 20130924 (UTC), we stopped using at least the Flash and FennecAndroid symbols we have. See e.g. those nonsymbolized signatures for builds that had symbols before: https://crashstats.mozilla.com/report/list?signature=libmozalloc.so%400x9d2 https://crashstats.mozilla.com/report/list?signature=libxul.so%400xbb2c13%20%7C%20libxul.so%400xc10d15%20%7C%20libxul.so%400xc10cff%20%7C%20libxul.so%400xbf9775%20%7C%20libxul.so%400xbf976b%20%7C%20libxul.so%400xbfca6b%20%7C%20libxul.so%400x97bad7%20%7C%20libxul.so%400xbfcad5%20%7C%20libxul.so%400xba24b7%20%7C%20libxul.so%400xba2e51%20%7C%20libxul.so%400xba3ed5%20%7C%20libxul.so%400xba407b%20%7C%20... https://crashstats.mozilla.com/report/list?signature=npswf32_11_8_800_168.dll%400x343e9d https://crashstats.mozilla.com/report/list?signature=hang%20%7C%20WaitForMultipleObjectsEx%20%7C%20RealMsgWaitForMultipleObjectsEx%20%7C%20MsgWaitForMultipleObjects%20%7C%20npswf32_11_8_800_168.dll%400x35da0c
Reporter  
Comment 1•6 years ago


This hinders analysis of a lot of crashes atm.
Severity: normal → blocker
Reporter  
Comment 2•6 years ago


The first npswf32_11_8_800_168.dll@0x343e9d report affected seems to have been submitted at 20130924T18:22:38+00:00  so I think very slightly before is when this all started.
Reporter  
Comment 3•6 years ago


Also, https://crashstats.mozilla.com/report/list?signature=F1398665248_____________________________ which is the symbolized version, was seen the last time at 20130924T21:08:08+00:00 solarce said on IRC that RabbitMQ was rolled out on that day "between 11:30am and 3:30pm" (Pacific, UTC7) and that fits the timeframe of the issues in this bug.
Comment 4•6 years ago


> % svn diff Index: modules/socorro/files/prod/etcsocorro/processor.ini ===================================================================  modules/socorro/files/prod/etcsocorro/processor.ini (revision 75760) +++ modules/socorro/files/prod/etcsocorro/processor.ini (working copy) @@ 347,7 +347,7 @@ # name: processor_symbols_pathname_list # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces) # converter: socorro.processor.legacy_processor.create_symbol_path_str  #processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g" + processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g" # name: stackwalk_command_line # doc: the template for the command to invoke minidump_stackwalk bburton@althalus [09:13:31] [~/code/mozilla/sysadmins/puppet/trunk] > % svn ci m "fixing regression in processor config for socorro prod, bug 921042" Sending trunk/modules/socorro/files/prod/etcsocorro/processor.ini Transmitting file data . Committed revision 75771.
Assignee: nobody → bburton
Comment 5•6 years ago


Missing some symbols! > % svn diff Index: modules/socorro/files/prod/etcsocorro/processor.ini ===================================================================  modules/socorro/files/prod/etcsocorro/processor.ini (revision 75771) +++ modules/socorro/files/prod/etcsocorro/processor.ini (working copy) @@ 347,7 +347,7 @@ # name: processor_symbols_pathname_list # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces) # converter: socorro.processor.legacy_processor.create_symbol_path_str  processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g" + processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo # name: stackwalk_command_line # doc: the template for the command to invoke minidump_stackwalk bburton@althalus [09:18:17] [~/code/mozilla/sysadmins/puppet/trunk] > % svn ci m "fixing regression in processor config for socorro prod, bug 921042" Sending trunk/modules/socorro/files/prod/etcsocorro/processor.ini Transmitting file data . Committed revision 75772.
Comment 6•6 years ago


Ok, I guess now we will need to reprocess crashes. Are the zte ones meant to be included?
Group: mozillacorporationconfidential
Reporter  
Comment 7•6 years ago


(In reply to Laura Thomson :laura from comment #6) > Ok, I guess now we will need to reprocess crashes. > > Are the zte ones meant to be included? Yes. Why shouldn't they?
Comment 8•6 years ago


Ok so here's what we need to reprocess: diff the config we have now (thanks solarce) vs the defaults, and reprocess anything that uses the symbols we just added back, that came in on 9.24 or later.
Comment 9•6 years ago


> % svn diff Index: modules/socorro/files/prod/etcsocorro/processor.ini ===================================================================  modules/socorro/files/prod/etcsocorro/processor.ini (revision 75772) +++ modules/socorro/files/prod/etcsocorro/processor.ini (working copy) @@ 347,7 +347,7 @@ # name: processor_symbols_pathname_list # doc: comma or space separated list of symbol files for minidump_stackwalk (quote paths with embedded spaces) # converter: socorro.processor.legacy_processor.create_symbol_path_str  processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo + processor_symbols_pathname_list="/mnt/socorro/symbols/symbols_ffx,/mnt/socorro/symbols/symbols_sea,/mnt/socorro/symbols/symbols_tbrd,/mnt/socorro/symbols/symbols_mob,/mnt/socorro/symbols/symbols_penelope,/mnt/socorro/symbols/symbols_sbrd,/mnt/socorro/symbols/symbols_camino,/mnt/socorro/symbols/symbols_os,/mnt/socorro/symbols/symbols_solaris,/mnt/socorro/symbols/symbols_opensuse,/mnt/socorro/symbols/symbols_ubuntu,/mnt/socorro/symbols/symbols_fedora,/mnt/socorro/symbols/symbols_adobe,/mnt/socorro/symbols/symbols_b2g,/mnt/socorro/symbols/symbols_geeksphone,/mnt/socorro/symbols/symbols_tclpartner,/mnt/socorro/symbols/symbols_zte,/mnt/socorro/symbols/symbols_leo" # name: stackwalk_command_line # doc: the template for the command to invoke minidump_stackwalk bburton@althalus [10:06:53] [~/code/mozilla/sysadmins/puppet/trunk] > % svn ci m "fixing missing closing double quote in symbols_pathname, in processor config for socorro prod, bug 921042" Sending trunk/modules/socorro/files/prod/etcsocorro/processor.ini Transmitting file data . Committed revision 75778.
Updated•6 years ago

Assignee: bburton → sdeckelmann
Reporter  
Comment 10•6 years ago


From what I saw on IRC, it looks like the default set is this: /mnt/socorro/symbols/symbols_ffx, /mnt/socorro/symbols/symbols_sea, /mnt/socorro/symbols/symbols_tbrd, /mnt/socorro/symbols/symbols_sbrd, /mnt/socorro/symbols/symbols_os And we have been using this since the start of this issue. So, what we have not been using in this time is this set: /mnt/socorro/symbols/symbols_mob, /mnt/socorro/symbols/symbols_penelope, /mnt/socorro/symbols/symbols_camino, /mnt/socorro/symbols/symbols_solaris, /mnt/socorro/symbols/symbols_opensuse, /mnt/socorro/symbols/symbols_ubuntu, /mnt/socorro/symbols/symbols_fedora, /mnt/socorro/symbols/symbols_adobe, /mnt/socorro/symbols/symbols_b2g, /mnt/socorro/symbols/symbols_geeksphone, /mnt/socorro/symbols/symbols_tclpartner, /mnt/socorro/symbols/symbols_zte, /mnt/socorro/symbols/symbols_leo So, we need to reprocess everything where those are used. This means at least all in FennecAndroid, WebappRuntimeMobile, and B2G products; all Flash plugin reports (as all others are few, that might just mean all plugin reports); and all Linux reports, submitted since 20130924T18:22:38+00:00
Updated•6 years ago

Group: mozillacorporationconfidential
Assignee  
Comment 11•6 years ago


The problem arose because of a configuration setting that was incorrectly commented out during our reconfiguration which enabled RabbitMQ. The setting fell back to using default directories we specify in our source tree, leaving out things like our ZTE, linux and various other important symbols. The problem affected about ~120k crashes. We started reprocessing at about 12:38pm PT. Processing concluded at about 1:45pm PT, and I started the matview backfill shortly thereafter. 9/25 is now complete. 9/24 has been started.
Assignee  
Comment 12•6 years ago


Backfill now complete.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution:  → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•