558880 - interesting modules files are zero length

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[chris hofmann]

the firefox "interesting-modules" files seem to be zero length starting around 3/31 and continuing with the latest updates under crash_analysis.  See:

http://people.mozilla.com/crash_analysis/20100331/

to 

http://people.mozilla.com/crash_analysis/20100412/


   0 Apr 12 12:54 20100411_Firefox_3.5.9-interesting-modules.txt20100411_Firefox_3.5.9-interesting-modules-with-versions.txt

Comment 1

[chris hofmann]

the 3.6.3 versions of the files on 3/30 and 3/29 are also zero length, so the problem may have started sometime after 3/28

Comment 2

[Aravind Gottipati [:aravind]]

Something wrong with the python code that generates these reports.  Here is the error I have from the logs.

[processor@cm-breakpad02 ~]$ python26 /data/breakpad/crash-data-tools/per-crash-interesting-modules.py -p Firefox -r 3.5.9 -f /tmp/Firefox_3.5.9.tar > /tmp/Firefox_3.5.9-interesting-modules.txt
Traceback (most recent call last):
  File "/data/breakpad/crash-data-tools/per-crash-interesting-modules.py", line 124, in <module>
    for libname, version in generate_modules_or_addons(crash):
  File "/data/breakpad/crash-data-tools/per-crash-interesting-modules.py", line 76, in generate_modules_or_addons
    addrend, unknown = dump_line.split("|")
ValueError: need more than 4 values to unpack
[processor@cm-breakpad02 ~]$

Comment 4

[Benjamin Smedberg]

Does this script live in a repo somewhere? And do you know what dump is being processed when the failure happens? jimb or I can probably hook this up pretty quick.

Comment 5

[Aravind Gottipati [:aravind]]

(In reply to comment #4)
> Does this script live in a repo somewhere? And do you know what dump is being
> processed when the failure happens? jimb or I can probably hook this up pretty
> quick.

Yes, I pulled it from http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/

I don't know which uuid in the tens of thousands that are in the tar file is causing the script to choke.  I can upload the tar file, if you want to play with it, but its fairly sensitive data.. so I'd be careful with it.  Let me know how you want to proceed and we can figure something out.

Comment 6

[Jim Blandy :jimb]

Aravind, if I could get a copy of an input file before 3/28 and a current one, I can take a look at this.  Perhaps you could mail them to me at my Mozilla e-mail address.

Comment 7

[Jim Blandy :jimb]

Based on the error message and the script, I wonder if the format of the machine-readable output changed.

Comment 8

[Benjamin Smedberg]

I'd like a copy of the current one, at least, however you want to post it.

Comment 9

[Aravind Gottipati [:aravind]]

Sent you guys a tar file with the info.

Comment 10

[Benjamin Smedberg]

The offending line being parsed is:

Module|olek8r4u.dll|6.0.6000.16386|\xc2\xeb\x17\x04J\xb6:\xbaT\xf3\xef\xe8Y\x90\x86\xaa\xe5\x16n\xb1\x80\x85\t\x12!\x16\x0f\x98\xf8\x89\x16"\x96\xd4\x84\x88\xea\xe3\r\r\x1b\xca\x85*^h\xf5\xdc

What's happening is that there is an embedded newline in the module description:

Module|olek8r4u.dll|6.0.6000.16386|\\xc2\\xeb\\x17\\x04J\\xb6:\\xbaT\\xf3\\xef\\xe8Y\\x90\\x86\\xaa\\xe5\\x16n\\xb1\\x80\\x85\\t\\x12!\\x16\\x0f\\x98\\xf8\\x89\\x16"\\x96\\xd4\\x84\\x88\\xea\\xe3\\r\\r\\x1b\\xca\\x85*^h\\xf5\\xdc\n\\xd9\\xf4}j\\x1d7\\xe39o\\x1f\\xc5\\xc4\\xa6x\\x8ba\\xe8\\xd6K\\x89H\\xe1\\xff\\xe7\\xf5\\xf0Y\\xfd\\xf5\\xdbu\\x0c\\x07\\x86\\xed|29E0B04FCCBE47EB86A6C819E8B89D051|0x00f60000|0x00ff2fff|0\n

This is a bug in dump_syms, it should do something smarter with the embedded newline, although I'm not sure what yet.

Comment 11

[Benjamin Smedberg]

Filed http://code.google.com/p/google-breakpad/issues/detail?id=383 and posted a patch there.

Comment 12

[K Lars Lohn [:lars] [:klohn]]

That bad module line looks to be the same one that broke Socorro Processor in Bug 556690 and in the Socorro UI in Bug 556888.  We received a bunch of crashes like this one at the beginning of the month.

Comment 13

[Jim Blandy :jimb]

Patch mentioned in comment 11 landed upstream as r571.

Comment 14

[chris hofmann]

still time to get this in to the 1.6.1 release that's being pushed tonight?

is the confidence in the patch good enough at this point to put it into production?

the number of bugs held up needing some module correlation data is growing so getting this as soon as possible would be great.

Comment 15

[Benjamin Smedberg]

This is a change to minidump_stackwalk which can be put into production independently of any socorro changes. If we want to make the correlation scripts ignore the existing bad data, that would be a different patch.

Comment 16

[chris hofmann]

I guess the question then is do we think the minidump_stackwalk change is ready for production?  seems better to just patch things there than to possibly continue to stumble on places that choke on the bad data.

Comment 17

[Jim Blandy :jimb]

Ben's patch from comment 11 looks extremely innocent to me.  I think it could safely be applied locally.

Comment 18

[Jim Blandy :jimb]

Other changes made upstream (r571, in particular) fail to compile on RHEL 5.4.  bug 560377 refers to the patch to fix that problem.  Marking that as blocking this, since the fix here is to re-import upstream.

Comment 19

[Aravind Gottipati [:aravind]]

This code is now running in production.

Comment 20

[Jim Blandy :jimb]

Chris, can you verify that the problem is fixed? It looks to me like the interesting-modules.txt lists are no longer being created as empty files, so I'd like to close this bug.

Comment 21

[chris hofmann]

yeah, looks fixed after http://people.mozilla.com/crash_analysis/20100423/