503185 - [Flash Player] Inclusion of memory offsets to symbols in crash reports would be helpful

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Charles]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.0.11) Gecko/2009060214 Firefox/3.0.11
Build Identifier: 

Adobe Flash Player is requesting that crash reports on Mac and Windows provide the offset to the symbols being captured.

For example, while Flash_EnforceLocalSecurity is not helpful to us, Flash_EnforceLocalSecurity + 60892 is helpful, and this is how the Apple crash reporting system saves the data.

Reproducible: Always

Actual Results:  
No offset with symbols in crash data reports

Expected Results:  
Provide this offset so we can have a better chance of understanding the root cause of the bug.

Comment 1

[(not currently active) Ted Mielczarek]

This data is actually available, Socorro just doesn't display it. Note in the raw dump:
http://crash-stats.mozilla.com/report/index/7250b49a-73aa-455d-91a8-91f822090707
18|3|Flash Player|Flash_EnforceLocalSecurity|||0x8151

So that's Flash_EnforceLocalSecurity + 0x8151.

Comment 2

[Michelle Sintov]

(In reply to comment #1)
The system that collates the crashes and determines the #1, #2, #3, etc. crashes needs to take into account that offset data. Right now, "Flash_EnforceLocalSecurity" is the top Mac crasher, but we need to instead see each offset as a different crash.

Comment 3

[(not currently active) Ted Mielczarek]

I think maybe it'd just be better if we got rid of the useless Flash Player symbols we have on the server currently. They're clearly not useful for actual crash analysis.

Comment 4

[Michelle Sintov]

Probably doesn't need to be said, but.... The Flash Player team is able to symbolize the crash logs given the hex addresses and the symbols + offsets. With Flash_EnforceLocalSecurity alone, we get nothing.

Comment 5

[Frank Griswold [:griswolf] [:fgriswold]]

Now, the socorro code prefers a plain function name. If that isn't available, it tries sourcefile#lineno. If sourcefile isn't available it gives module (if known)@<instruction>

We could add "@<instruction>" (if known) to the function name with reasonable ease if we do it always. Is that acceptable for all functions? If not, it gets much more complex for us to provide it only sometimes. Bear in mind that on 64-bit platforms, the address can get pretty large.

Note to socorro team: The place to add the instruction for functions is near line 298 in processor.py

Comment 6

[timeless]

for public export only frames, we want @Foo+0xOFF534

Comment 7

[Frank Griswold [:griswolf] [:fgriswold]]

comment #6 is too telegraphic to help me:

How can socorro recognize 'public export only frames'? 

What is the "@Foo" here?

Would it be good to show the instruction/address whenever we know it, or do we need to show it _only_ for 'public export only frames' to avoid obfuscating more important details?

Are we sure that the instruction field we receive is actually an offset from the start of the function? If not (or not always), might we confuse people with the suggested format?

Remember, socorro has only the information that is in the raw dump. For http://crash-stats.mozilla.com/report/index/7250b49a-73aa-455d-91a8-91f822090707 that means we have this line for the frame in question.

18|3|Flash Player|Flash_EnforceLocalSecurity|||0x8151 

which translates to:
thread_num = 18
frame_num = 3
module_name = Flash Player
function = Flash_EnforceLocalSecurity
source is unknown
source_line is unknown
instruction = 0x8151

Comment 8

[timeless]

if you don't have symbols for a file then it is a public export.
symbols mean one of two things:
()s in the function name, or a filename in the filename field

by way of example:
18|9|libSystem.B.dylib|_pthread_start|||0x140 

no ()s in the function name, and not filename in the filename field
=> public export. show as @ _pthread_start + 0x140

0|12|libmozjs.dylib|js_Invoke|hg:hg.mozilla.org/releases/mozilla-1.9.1:js/src/jsinterp.cpp:a625a31a0ad1|1386|0x27

no ()s in the function name, but hg:... in the filename
=> not a public export. show as @ js_Invoke

0|33|XUL|nsTimerEvent::Run()|hg:hg.mozilla.org/releases/mozilla-1.9.1:xpcom/threads/nsTimerImpl.cpp:a625a31a0ad1|512|0x7

()s in the function name => not a public export.

Show as @ nsTimerEvent::Run()

You can use pthread_start@0x140 instead of + I suppose, but historically for talkback we used plus because that made sense (and is what the debuggers actually use).

Comment 9

[Jesse Ruderman]

When we have full symbol information, I'd prefer for +offset to not be part of the signature used for collation, since it would interfere with comparing across versions and operating systems.  For example, nsScriptLoader::StartLoad+0x3 on Windows is probably the same as nsScriptLoader::StartLoad+0x0 on Mac.

When we don't have full symbol information (like for Flash), I think we should include the offset as part of the collated signature, so we at least have some idea of what the most common Flash crashes are.  And it should be an offset from the beginning of the DLL, not from something like Flash_EnforceLocalSecurity.

Comment 10

[Jesse Ruderman]

It might even be sensible to include the version of the plugin as part of the signature, but that would depend on bug 503181.

Comment 11

[(not currently active) Ted Mielczarek]

I'm going to repeat comment 3 and say that we should just delete the useless Flash Player symbols we have on the symbol server. Unfortunately Smokey uploaded them all, so I don't have permission to delete them. Smokey?

If we delete them, then crashes will just show up as Flash Player@0xABCDEF, with the relative offset into the library.

Comment 12

[chris hofmann]

aravind can do the removal too if we give him a list of directory/files

Comment 13

[(not currently active) Ted Mielczarek]

just rm -rf /mnt/netapp/breakpad/symbols_os/Flash\ Player/

Comment 14

[Aravind Gottipati [:aravind]]

(In reply to comment #13)
> just rm -rf /mnt/netapp/breakpad/symbols_os/Flash\ Player/

Done.

Comment 15

[(not currently active) Ted Mielczarek]

I'm going to call this fixed for now. New crash reports should start showing up with simply offsets into the Flash Player module (just like the windows crashes in npswf32.dll).

Comment 16

[Michelle Sintov]

A big thank you from the Flash Player team!

Comment 17

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #11)
> I'm going to repeat comment 3 and say that we should just delete the useless
> Flash Player symbols we have on the symbol server. Unfortunately Smokey
> uploaded them all, so I don't have permission to delete them. Smokey?

Sorry for the belated reply; I'm just getting to this, er, last week's bug changes now :-(