532011 - Flash crash [@ NPSWF32.dll@0xca950 ]

Status: RESOLVED INVALID

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Description

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

This is currently the #8 topcrash for 3.6b4. However it's also the #10 topcrash on 3.5.5 so it's unlikely a regression on our end.

All crashes with this signature (in the past week):

http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=NPSWF32.dll%400xca950&date=&range_value=1&range_unit=weeks&do_query=1&signature=NPSWF32.dll%400xca950

Just the 3.6b4 crashes:

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.6b4&query_search=signature&query_type=exact&query=NPSWF32.dll%400xca950&date=&range_value=1&range_unit=weeks&do_query=1&signature=NPSWF32.dll%400xca950

Comment 1

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

It appears this crash always happens as we are trying to stop a flash instance. I looked at 3 crash stacks and they all looked like this:

NPSWF32.dll!6660a950()  
NPSWF32.dll!66610c39()  
NPSWF32.dll!66612fba()  
NPSWF32.dll!66615c88()  
NPSWF32.dll!6677236d()  
NPSWF32.dll!6676fcd0()  
NPSWF32.dll!66771356()  
NPSWF32.dll!667715e6()  
NPSWF32.dll!66631275()  
gdi32.dll!_IcmDeleteLocalDC@12()  + 0x21 bytes  
NPSWF32.dll!66542323()  
NPSWF32.dll!66542323()  
NPSWF32.dll!6654ead2()  
NPSWF32.dll!6654eafc()  
NPSWF32.dll!66586dce()  
NPSWF32.dll!66568616()  
NPSWF32.dll!6668022a()  
NPSWF32.dll!666793b0()  
NPSWF32.dll!66679837()  
NPSWF32.dll!66673ed2()  
xul.dll!nsNPAPIPluginInstance::Stop() Line 1178
xul.dll!nsPluginNativeWindow::CallSetWindow() Line 103
xul.dll!DoStopPlugin() Line 2249
xul.dll!nsStopPluginRunnable::Run()
xul.dll!nsThread::ProcessNextEvent()
xul.dll!nsBaseAppShell::Run()


The only thing that differed was how we got to xul.dll!DoStopPlugin. The stacks inside NPSWF32.dll were all exactly the same (differing only by where NPSWF32.dll was loaded, so by the first 4 address numbers).


Somehow MSVC is actually able to bring up the NPSWF32.dll code, though of course without symbols. The code that's crashing is doing the following:


push        ebx  
lea         ebx,[esi+2ACh] 
mov         dword ptr [esp+10h],81h 
mov         edi,dword ptr [ebx] 
cmp         edi,ebp 
je          6B7FA96C 
mov         edi,edi 
mov         ebp,dword ptr [edi+0B8h]   ; crashing here!
mov         ecx,edi 
call        6B7E3940 

And the following register values:
EAX = 00000000 EBX = 0CF51394 ECX = 0BD0E028 EDX = 6BA2FD34 ESI = 0CF510A0 
EDI = 80000000 EIP = 6B7FA950 ESP = 0028F23C EBP = 00000000 EFL = 00010286 

So it seems like we nullcheck an object, grabbing a member variable and then make a non-virtual this-call. However it appears that the object pointer is 0x80000000 and so we pass the nullcheck but fail when trying to load the member variable.

Unfortunately this probably doesn't help much without symbols :(

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comparing 3.5.5 data and 3.6b4 data for the last 3 days (used a shorter period to avoid strangeness around thanksgiving):

3.5.5 crashes: 2405
3.6b4 crashes: 395

Comparing that to total number of crashes for 11/30
3.5.5 crashes: 99896
3.6b4 crashes: 11574


If they ratio were exactly the same, we'd only see 267 crashes on 3.6b4. So we're slightly higher in 3.6b4 than we are on 3.5.5

Comment 3

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

And data for 3.0.15:

Crashes with this signature over past 3 days: 780
Total number of crashes on 11/30: 3873

So 3.6b4 is actually crashing a good bit less than 3.0.15. Though possibly that could be because 3.0.15 has gotten very stable after so many dot-releases, and so the total number of crashes is much lower relative to number of users.

Comment 4

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Charles: I don't think I can get much further here. All data is pointing to this being a flash-specific crash that has been around for several releases of firefox.

Is this something you can look into on the adobe side?

Is there any way I can help?

Chofmann: Can you see if there's any correlation with any specific urls here?

Comment 5

[chris hofmann]

here is what I get when looking at data from yesterday

distribution of all versions where the NPSWF32.dll@0xca950 crash was found on 20091130-crashdata.csv
759     0.624178        3.5.5
249     0.20477         3.0.15
160     0.131579        3.6b4
7       0.00575658      3.6b3
7       0.00575658      3.5.2
5       0.00411184      3.5.3
4       0.00328947      3.5.4
3       0.00246711      3.6b1
[1-3 crashes on many other firefox releases]

pretty wide distribution across all windows versions

os breakdown
461     0.379112        Windows NT5.1.2600 Service Pack 3
261     0.214638        Windows NT6.0.6002 Service Pack 2
178     0.146382        Windows NT5.1.2600 Service Pack 2
121     0.0995066       Windows NT6.1.7600
120     0.0986842       Windows NT6.0.6001 Service Pack 1
43      0.0353618       Windows NT6.0.6000
18      0.0148026       Windows NT6.1.7100


The highest concentration of these crashes is on the sites listed below.  Seems like a lot of international sites, specifically fr and es-mx content.

crash count for 2009-11-30
domains of sites
 316 http://www.jeuxvideo.com
 101 http://webmail.laposte.net
  83 \N//
  27 http://www.google.fr
  26 http://www.naruto-mx.com
  25 http://www.footmercato.net
  19 http://www.facebook.com
  16 http://www.ultimate-guitar.com
  16 http://www.tresorquest.com
  13 http://www.fors60.com
  13 http://nl.ptg.be
  13 http://narutodirect.blog.jeuxvideo.com
  12 http://www.jeanmarcmorandini.com
  11 http://www.comlive.net

specific pages.   only a bit of stanitization was required on a a few of these.  most of the content appears to be public and not behind any login/auto or personal info in the query string.

  39 http://www.jeuxvideo.com/etajvbis.htm
  15 http://www.tresorquest.com/index.php
  14 http://www.jeuxvideo.com/articles/0001/00011908-blood-bowl-test.htm
  14 http://www.footmercato.net/
  11 http://www.naruto-mx.com/
   8 http://www.jeuxvideo.com/pc.htm
   8 http://www.jeuxvideo.com/news.htm
   8 http://webmail.laposte.net/webmail/fr_FR/inbox.html?PAGE=1
   6 http://www.jeuxvideo.com/x360-xbox-360.htm
   6 http://www.basketfrance.com/
   6 http://nl.ptg.be/fidelity/
   5 http://www.signification-prenom.com/
   5 http://www.naruto-mx.com/streaming/episode-naruto-shippuden-137-vostfr-vf.html
   5 http://www.jeuxvideo.com/gaming_live.htm
   5 http://www.google.fr/
   5 http://www.facebook.com/home.php?ref=home
   5 http://fr.ptg.be/fidelity/
   4 http://www.naruto-mx.com/streaming/episodes-naruto-shippuden-vostfr-vf.html
   4 http://www.jeuxvideo.com/forums/0-18353-0-1-0-1-0-call-of-duty-modern-warfare-2.htm
   4 http://www.jeuxvideo.com/commentaires/9-0-38927-1-0-1-0-0.htm#commentaires
   4 http://www.jeuxvideo.com/cheats.htm
   4 http://www.jeanmarcmorandini.com/
   4 http://webmail.laposte.net/webmail/fr_FR/delete_submit.html
   3 http://www.lephoceen.fr/
   3 http://www.jeuxvideo.com/ps3-playstation-3.htm
   3 http://www.jeuxvideo.com/news/2009/00038931-modern-warfare-2-prime-pour-ses-ventes.htm
   3 http://www.jeuxvideo.com/news/2009/00038929-images-de-final-fantasy-xiii.htm
   3 http://www.jeuxvideo.com/news/2009/00038927-ce-qui-vous-fait-acheter-un-jeu.htm
   3 http://www.jeuxvideo.com/news/2009/00038923-meilleures-ventes-de-jeux-en-france-ezio-fait-le-beau.htm
   3 http://www.jeuxvideo.com/le-cliq/2009/30-11-2009.htm
   3 http://www.jeuxvideo.com/jeux/playstation-3-ps3/00028271-colin-mcrae-dirt-2.htm
   3 http://www.jeuxvideo.com/forums/0-21115-0-1-0-1-0-wwe-smackdown-vs-raw-2010.htm
   3 http://www.jeuxvideo.com/dossiers/00011758/les-jeux-de-noel-2009.htm
   3 http://www.jeuxvideo.com/articles/0001/00011889-runaway-a-twist-of-fate-test.htm
   3 http://www.fors60.com/applications.php?add_application=&page=3&search=&test=&download=
   3 http://www.footmercato.net/mercato-l1-libres-l-ete-prochain,42011_article42011.html
   3 http://webmail.laposte.net/webmail/fr_FR/inbox.html

Comment 6

[Josh Aas]

(In reply to comment #1)

xul.dll!nsPluginNativeWindow::CallSetWindow() Line 103

This frame from the stack in comment 1 is almost certainly bogus. Subtract it and the stack makes much more sense.

Comment 7

[Jet Villegas (inactive)]

Picking this up. Can we get the stack trace against the submitted symbols?

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Jethro: What "submitted symbols"? My impression was that we're still at the unfortunate point of us not being able to share minidumps with you guys, and you guys not being able to share symbols with us.

I know there was discussions about movement in either direction, but I haven't heard anything about actual progress.

Have you heard otherwise, if so that'd be great news.

Comment 9

[Charles]

(In reply to comment #8)
> Jethro: What "submitted symbols"? My impression was that we're still at the
> unfortunate point of us not being able to share minidumps with you guys, and
> you guys not being able to share symbols with us.
> 
> I know there was discussions about movement in either direction, but I haven't
> heard anything about actual progress.
> 
> Have you heard otherwise, if so that'd be great news.

We have provided obfuscated symbols for various versions.  Please coordinate with Chris or Ben to see we can get more data from these crashes.

Comment 10

[Scoobidiver (away)]

With Flash debug symbols, if these crashes still exist, they will be tracked in another bug.
I close it as invalid.