412605 - collect list of extensions

Status: RESOLVED FIXED

(Socorro :: General, task, P1)

Description

[(not currently active) Ted Mielczarek]

Mossop is working on bug 366973, so we should get the server-side things in place to actually collect and store the list of extensions.

Comment 1

[(not currently active) Ted Mielczarek]

bug 366973 is checked in.  Mossop: can you elaborate on what format the client sends this data in?

Comment 2

[Dave Townsend [:mossop]]

There are two annotations. "Add-ons" is a comma separated string of add-ons. It will contains all enabled add-ons (extensions and themes). Each part of the string will be <addon-id>:<addon-version>. The second annotation is "Theme" which contains the skin name of the theme, e.g. "classic/1.0" for the default theme.

Comment 3

[chris hofmann]

we could really use this to get a handle on many crashes in the top20-top40 list.  is it possible to bump the priority and make it blocking for fx3?

this bug would also need to cover getting the info into the reporting system to make it valuable, or should we open another for that?

Comment 4

[Benjamin Smedberg]

I don't think it should block: it's certainly not a regression against FF2.

Why don't we focus instead on doing statistical regression analysis against the list of process DLLs? That will give us numbers for crashes that are related to installed plugins or extensions that ship with binary components, and that data is already available in the database. I'm trying to find the bug# for that, but can't... I'm pretty sure I filed it in google code a long time ago.

Comment 5

[chris hofmann]

analysis of the process list might work.  that's basically what I think most of us are trying to do by eye-balling it now.  what would the reporting look like?

don't know if this is possible without a lot of maintenance, but
it would be great of the process list could be reorganized to "hide" all the base firefox componenents, and all the "required OS components" to just leave these optional components visible in the report

 + - firefox components
 + - OS components
 - - other plugins, extensions, and programs
      vmsfdm.dll
      ...
      ...

Comment 6

[Benjamin Smedberg]

That's not practical, really... we don't know which components belong to Firefox, the OS, or etc without hand-maintaining a list. But I don't think we need that; instead, you can do a regression analysis of "people experiencing this crash signature" versus "the general population" to come up with a list of "DLLs that correlate strongly with this crash signature". (per-OS... it doesn't make much sense to try and correlate this across OSes, because of the differences in DLL naming). This will automatically exclude Firefox/system DLLs from consideration, because they are present in 100% of cases so there will be no correlation.

We should probably cache the DLL instances in the general population so that each individual crash report doesn't have to perform that full-scan query again. Morgamic, do you have thoughts on how we could do this effectively?

You could correlate against particular versions of DLLs, but I think you're less likely to get valuable data out of that by default... you'd only really need to do it if you wanted to discover things like "this crash doesn't happen with Flash 9R1 but does with Flash 9R2" or "This crash only happens when Windows Vista security update X was installed".

Comment 7

[Michael Morgan [:morgamic]]

Ken would you be able to help with this?

Comment 8

[ken kovash]

I should be able to help with any statistical regression analysis.  I'm copying Polvi for some assistance.

Comment 9

[chris hofmann]

one thing that we could get out of the .dll version analysis would be a faster way to spot situations like bug 380015.    we should also be designing the reporting to tell us about .dll version mismatches as well if possible.

Comment 10

[(not currently active) Ted Mielczarek]

We would need to have a list of known modules + version numbers per-release to make that happen. Wouldn't be hard for actual releases, just wouldn't work for nightlies.

Comment 11

[chris hofmann]

ted suggested splitting off the process list analysis part of this discussion into a new bug.  I filed https://bugzilla.mozilla.org/show_bug.cgi?id=423968

Comment 12

[Mike Schroepfer]

Would be good to get this done..

Comment 13

[ken kovash]

in regards to the data and available variables that chofmann and ben previously referenced, would someone be able to provide me with a sample of the data (something as a .txt, .csv or .xls)?  if I can *see* what the data looks like, I'll have a better understanding of what needs to be done from a statistical perspective.  thanks!

Comment 14

[chris hofmann]

https://bugzilla.mozilla.org/show_bug.cgi?id=422018 has analyis of one of the easy cases where the suspect .dll is in the "stack signature".

https://bugzilla.mozilla.org/show_bug.cgi?id=382356 is an example of a bit harder case where examination of the process data was needed to see that idmmzcc.dll was at the root of the problem.

click on
http://crash-stats.mozilla.com/report/index/d1550640-f0c3-11dc-8dbd-001a4bd43ef6
then the "modules" tab  to see the process list.

it looks something like this with a combination of firefox and system .dlls:

firefox.exe  	1.9.0.2988  	3096B91BE19044BDA24A7BE8D7F4B67C2  	firefox.pdb
idmmzcc.dll 	4.0.0.1 		
dbghelp.dll 	5.1.2600.2180 	39559573E21B46F28E286923BE9E6A761 	dbghelp.pdb
mozcrt19.dll 	8.0.0.0 	2B3B9EA89E5F40DC9D16DD2EDE88183F1 	mozcrt19.pdb
nspr4.dll 	4.7.1.0 	AFF27D393D1C46A89418F67146C19C931 	nspr4.pdb
plds4.dll 	4.7.1.0 	752791B6A24E43C98B1103F8DE4424D21 	plds4.pdb
plc4.dll 	4.7.1.0 	D35204417A204225825D7F5527CA23951 	plc4.pdb
js3250.dll 	4.0.0.0 	A358952855574E0CB6129FB898E57C452 	js3250.pdb
browserdirprovider.dll 	1.9.0.2988 	71045A99CD9A4583AF132796F7C667823 	browserdirprovider.pdb
sqlite3.dll 	3.5.4.1 	65D1230C228742398A741C7AA212F3A81 	sqlite3.pdb
nss3.dll 	3.12.0.0 	A582E1A60BA841B6AED236268CDB49DC1 	nss3.pdb
nssutil3.dll 	3.12.0.0 	E2AD3F37F8144247B91361095CA1B7CA1 	nssutil3.pdb
ssl3.dll 	3.12.0.0 	FA3E32FEAAF54296AF3D8B15A1BB33051 	ssl3.pdb
smime3.dll 	3.12.0.0 	4CC45BF21C6D4D97B0239345D0291CD11 	smime3.pdb
xul.dll 	1.9.0.2988 	5C2A97CA0155469CA849E70CA87FB314c 	xul.pdb
xpcom.dll 	1.9.0.2988 	FAD69927892F48D9A6F956389512E4783 	xpcom.pdb
ws2help.dll 	5.1.2600.2180 	537CE830EFE94FE3A92C95153BDB71462 	ws2help.pdb

Comment 15

[ken kovash]

Attachment: sample data set

Attached is a sample of the format and file type that would be helpful [for the first steps in figuring out how we might analyze this data].  I'm assuming morgamic/arvind can write a script that dumps out the data this way (a .csv or .txt file would also work).   ideally, the data set would include a row for each and every crash (i.e., across all Fx versions), so I'm okay with a data set that has hundreds of thousands (or millions) of instances/rows, and I'll start to play with that in a statistical program to see if we can make one pass to identify common .exe's and .dlls not connected with *any* crash, then a second pass to try and connect .exe's and .dll's that have strong connections to individual crashes.

This is the best way for me to apply some fancy statistical analysis to the data that we have.

If we can make those kind of connections first, then the next step will be to script something up in the socorro reporting system that follows the same analysis steps.   the summary reporting system will be working toward something along the lines of: "for crash X and platform Y, 80% have a flash dll, 20% quicktime, 3% java, etc."

Where we have seen a particular .dll associated with a crash, the correlation will be 100% for that stack signature.

Comment 16

[(not currently active) Ted Mielczarek]

I don't think this should block, the ongoing work here belongs in bug 423968.

Comment 17

[Damon Sicore (:damons)]

-'ing based on comment #16.

Comment 18

[chris hofmann]

this would be good to get for analyzing firefox 3.6 data...   once the data is in the db we will need reports to look at it.  should that be another bug?

Comment 19

[K Lars Lohn [:lars] [:klohn]]

This is submitted for testing on staging.  A push to production should happen soon thereafter.

Comment 20

[K Lars Lohn [:lars] [:klohn]]

As of 2009/09/18, Socorro is saving information about extensions in the database for each crash.  Unfortunately, there is no UI to access this information.  If you have ideas on the types of reports that you'd like to see, please note them on the new Bug 518232.  This new bug is specifically about the API.

Comment 21

[K Lars Lohn [:lars] [:klohn]]

in my last comment:

s/API/UI