Closed Bug 558880 Opened 12 years ago Closed 12 years ago

interesting modules files are zero length

Categories

(Socorro :: General, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: chofmann, Assigned: jimb)

References

Details

the firefox "interesting-modules" files seem to be zero length starting around 3/31 and continuing with the latest updates under crash_analysis.  See:

http://people.mozilla.com/crash_analysis/20100331/

to 

http://people.mozilla.com/crash_analysis/20100412/


   0 Apr 12 12:54 20100411_Firefox_3.5.9-interesting-modules.txt20100411_Firefox_3.5.9-interesting-modules-with-versions.txt
the 3.6.3 versions of the files on 3/30 and 3/29 are also zero length, so the problem may have started sometime after 3/28
Something wrong with the python code that generates these reports.  Here is the error I have from the logs.

[processor@cm-breakpad02 ~]$ python26 /data/breakpad/crash-data-tools/per-crash-interesting-modules.py -p Firefox -r 3.5.9 -f /tmp/Firefox_3.5.9.tar > /tmp/Firefox_3.5.9-interesting-modules.txt
Traceback (most recent call last):
  File "/data/breakpad/crash-data-tools/per-crash-interesting-modules.py", line 124, in <module>
    for libname, version in generate_modules_or_addons(crash):
  File "/data/breakpad/crash-data-tools/per-crash-interesting-modules.py", line 76, in generate_modules_or_addons
    addrend, unknown = dump_line.split("|")
ValueError: need more than 4 values to unpack
[processor@cm-breakpad02 ~]$
Duplicate of this bug: 559063
Does this script live in a repo somewhere? And do you know what dump is being processed when the failure happens? jimb or I can probably hook this up pretty quick.
(In reply to comment #4)
> Does this script live in a repo somewhere? And do you know what dump is being
> processed when the failure happens? jimb or I can probably hook this up pretty
> quick.

Yes, I pulled it from http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/

I don't know which uuid in the tens of thousands that are in the tar file is causing the script to choke.  I can upload the tar file, if you want to play with it, but its fairly sensitive data.. so I'd be careful with it.  Let me know how you want to proceed and we can figure something out.
Aravind, if I could get a copy of an input file before 3/28 and a current one, I can take a look at this.  Perhaps you could mail them to me at my Mozilla e-mail address.
Based on the error message and the script, I wonder if the format of the machine-readable output changed.
I'd like a copy of the current one, at least, however you want to post it.
Sent you guys a tar file with the info.
Blocks: 542201
The offending line being parsed is:

Module|olek8r4u.dll|6.0.6000.16386|\xc2\xeb\x17\x04J\xb6:\xbaT\xf3\xef\xe8Y\x90\x86\xaa\xe5\x16n\xb1\x80\x85\t\x12!\x16\x0f\x98\xf8\x89\x16"\x96\xd4\x84\x88\xea\xe3\r\r\x1b\xca\x85*^h\xf5\xdc

What's happening is that there is an embedded newline in the module description:

Module|olek8r4u.dll|6.0.6000.16386|\\xc2\\xeb\\x17\\x04J\\xb6:\\xbaT\\xf3\\xef\\xe8Y\\x90\\x86\\xaa\\xe5\\x16n\\xb1\\x80\\x85\\t\\x12!\\x16\\x0f\\x98\\xf8\\x89\\x16"\\x96\\xd4\\x84\\x88\\xea\\xe3\\r\\r\\x1b\\xca\\x85*^h\\xf5\\xdc\n\\xd9\\xf4}j\\x1d7\\xe39o\\x1f\\xc5\\xc4\\xa6x\\x8ba\\xe8\\xd6K\\x89H\\xe1\\xff\\xe7\\xf5\\xf0Y\\xfd\\xf5\\xdbu\\x0c\\x07\\x86\\xed|29E0B04FCCBE47EB86A6C819E8B89D051|0x00f60000|0x00ff2fff|0\n

This is a bug in dump_syms, it should do something smarter with the embedded newline, although I'm not sure what yet.
That bad module line looks to be the same one that broke Socorro Processor in Bug 556690 and in the Socorro UI in Bug 556888.  We received a bunch of crashes like this one at the beginning of the month.
Assignee: nobody → jim
Patch mentioned in comment 11 landed upstream as r571.
still time to get this in to the 1.6.1 release that's being pushed tonight?

is the confidence in the patch good enough at this point to put it into production?

the number of bugs held up needing some module correlation data is growing so getting this as soon as possible would be great.
This is a change to minidump_stackwalk which can be put into production independently of any socorro changes. If we want to make the correlation scripts ignore the existing bad data, that would be a different patch.
I guess the question then is do we think the minidump_stackwalk change is ready for production?  seems better to just patch things there than to possibly continue to stumble on places that choke on the bad data.
Ben's patch from comment 11 looks extremely innocent to me.  I think it could safely be applied locally.
Assignee: jim → benjamin
Assignee: benjamin → jim
Other changes made upstream (r571, in particular) fail to compile on RHEL 5.4.  bug 560377 refers to the patch to fix that problem.  Marking that as blocking this, since the fix here is to re-import upstream.
Depends on: 560377
This code is now running in production.
Blocks: 561264
Chris, can you verify that the problem is fixed? It looks to me like the interesting-modules.txt lists are no longer being created as empty files, so I'd like to close this bug.
Status: NEW → ASSIGNED
yeah, looks fixed after http://people.mozilla.com/crash_analysis/20100423/
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.