516780 - Please upgrade to Flash Player 10.0.42.34 or later to fix crashes [@ NPSWF32.dll@0x77bd0 ] starting around around 31 Jul 2009, many at Farmtown

Status: RESOLVED FIXED

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Comment 1

[chris hofmann]

.3.5.1, released July 16th, 2009.

0   total crashes for NPSWF32.dll@0x77bd0 on 20090726-crashdata.csv
0   total crashes for NPSWF32.dll@0x77bd0 on 20090727-crashdata.csv
0   total crashes for NPSWF32.dll@0x77bd0 on 20090728-crashdata.csv
0   total crashes for NPSWF32.dll@0x77bd0 on 20090729-crashdata.csv
0   total crashes for NPSWF32.dll@0x77bd0 on 20090730-crashdata.csv

--> new verison of flash release

--> new versions of firefox v.3.5.2, released August 3rd, 2009

28   total crashes for NPSWF32.dll@0x77bd0 on 20090805-crashdata.csv
17   total crashes for NPSWF32.dll@0x77bd0 on 20090806-crashdata.csv
10   total crashes for NPSWF32.dll@0x77bd0 on 20090807-crashdata.csv
20   total crashes for NPSWF32.dll@0x77bd0 on 20090808-crashdata.csv
8   total crashes for NPSWF32.dll@0x77bd0 on 20090809-crashdata.csv


142   total crashes for NPSWF32.dll@0x77bd0 on 20090810-crashdata.csv
171   total crashes for NPSWF32.dll@0x77bd0 on 20090811-crashdata.csv
197   total crashes for NPSWF32.dll@0x77bd0 on 20090812-crashdata.csv
277   total crashes for NPSWF32.dll@0x77bd0 on 20090813-crashdata.csv
307   total crashes for NPSWF32.dll@0x77bd0 on 20090814-crashdata.csv
383   total crashes for NPSWF32.dll@0x77bd0 on 20090815-crashdata.csv
368   total crashes for NPSWF32.dll@0x77bd0 on 20090816-crashdata.csv
484   total crashes for NPSWF32.dll@0x77bd0 on 20090817-crashdata.csv
427   total crashes for NPSWF32.dll@0x77bd0 on 20090818-crashdata.csv
427   total crashes for NPSWF32.dll@0x77bd0 on 20090819-crashdata.csv
488   total crashes for NPSWF32.dll@0x77bd0 on 20090820-crashdata.csv
456   total crashes for NPSWF32.dll@0x77bd0 on 20090821-crashdata.csv
440   total crashes for NPSWF32.dll@0x77bd0 on 20090822-crashdata.csv
455   total crashes for NPSWF32.dll@0x77bd0 on 20090823-crashdata.csv
603   total crashes for NPSWF32.dll@0x77bd0 on 20090824-crashdata.csv
567   total crashes for NPSWF32.dll@0x77bd0 on 20090825-crashdata.csv
594   total crashes for NPSWF32.dll@0x77bd0 on 20090826-crashdata.csv
583   total crashes for NPSWF32.dll@0x77bd0 on 20090827-crashdata.csv
593   total crashes for NPSWF32.dll@0x77bd0 on 20090828-crashdata.csv
648   total crashes for NPSWF32.dll@0x77bd0 on 20090829-crashdata.csv
731   total crashes for NPSWF32.dll@0x77bd0 on 20090830-crashdata.csv
735   total crashes for NPSWF32.dll@0x77bd0 on 20090831-crashdata.csv
709   total crashes for NPSWF32.dll@0x77bd0 on 20090901-crashdata.csv
695   total crashes for NPSWF32.dll@0x77bd0 on 20090902-crashdata.csv
668   total crashes for NPSWF32.dll@0x77bd0 on 20090903-crashdata.csv
698   total crashes for NPSWF32.dll@0x77bd0 on 20090904-crashdata.csv
698   total crashes for NPSWF32.dll@0x77bd0 on 20090905-crashdata.csv
739   total crashes for NPSWF32.dll@0x77bd0 on 20090906-crashdata.csv
944   total crashes for NPSWF32.dll@0x77bd0 on 20090907-crashdata.csv
947   total crashes for NPSWF32.dll@0x77bd0 on 20090908-crashdata.csv
938   total crashes for NPSWF32.dll@0x77bd0 on 20090909-crashdata.csv
856   total crashes for NPSWF32.dll@0x77bd0 on 20090910-crashdata.csv
839   total crashes for NPSWF32.dll@0x77bd0 on 20090911-crashdata.csv
950   total crashes for NPSWF32.dll@0x77bd0 on 20090912-crashdata.csv
958   total crashes for NPSWF32.dll@0x77bd0 on 20090913-crashdata.csv
1014   total crashes for NPSWF32.dll@0x77bd0 on 20090914-crashdata.csv

Comment 2

[chris hofmann]

the facebook farmtown app is frequently assocated with this particular signature, but that isn't the only crashing signature on farmtown.

top crashing signatures for farmtown in the last 14 days
9811 NPSWF32.dll@0x77bd0
4539 NPSWF32.dll@0x77540
1765 NPSWF32.dll@0xbc897
1450 NPSWF32.dll@0xbbff7
877 Flash Player@0xe4a5f
571 NPSWF32.dll@0x77c41
429 RealDefWindowProcWorker

and there are around 3341 uniq signatures related to farm town crashes.

Comment 3

[chris hofmann]

on aug 10 when we see the jump above to 142 NPSWF32.dll@0x77bd crashes here was the distribution of all versions where the NPSWF32.dll@0x77bd0 crash was found 

 124 Firefox 3.5.2
   5 Firefox 3.5.1
   4 Firefox 3.5
   3 Firefox 3.0.10
   2 Firefox 3.0.6
   2 Firefox 3.0.1
   1 Firefox 3.0.9
   1 Firefox 3.0.11

on Sept 14 the distribution looks like

distribution of all versions where the NPSWF32.dll@0x77bd0 crash was found on 20090914-crashdata.csv
 699 Firefox 3.5.3
 209 Firefox 3.0.14
  69 Firefox 3.5.2
  18 Firefox 3.0.13
   5 Firefox 3.0.11
   4 Firefox 3.0.1
   3 Firefox 3.0.9
   2 Firefox 3.5
   2 Firefox 3.0.8
   2 Firefox 3.0.3
   1 Firefox 3.5.1



distribution of all versions where the NPSWF32.dll@0x77bd0 crash was found on 20090810-crashdata.csv
 124 Firefox 3.5.2
   5 Firefox 3.5.1
   4 Firefox 3.5
   3 Firefox 3.0.10
   2 Firefox 3.0.6
   2 Firefox 3.0.1
   1 Firefox 3.0.9
   1 Firefox 3.0.11

Comment 4

[Jim Mathies [:jimm]]

Pulling the patch may well address this, but it leaves open the original crashing problem in bug 493601. From the original bug: 

(In reply to comment #43)
> Just for the record:
> 0:000> !exploitable -v
> HostMachine\HostUser
> Executing Processor Architecture is x86
> Debuggee is in User Mode
> Debuggee is a live user mode debugging session on the local machine
> Event Type: Exception
> *** ERROR: Symbol file could not be found.  Defaulted to export symbols for
> C:\Program Files\Mozilla Firefox 3\xul.dll - 
> Exception Faulting Address: 0x4d75c9e
> First Chance Exception Type: STATUS_ACCESS_VIOLATION (0xC0000005)
> Exception Sub-Type: Data Execution Protection (DEP) Violation
> 
> Exception Hash (Major/Minor): 0x35022a3c.0x354a2a76
> 
> Stack Trace:
> <Unloaded_NPSWF32.dll>+0x135c9e
> <Unloaded_NPSWF32.dll>+0xdcb51
> xul!gfxSkipCharsIterator::GetSkippedOffset+0x9c1
> xul!XRE_main+0x1122
> Instruction Address: 0x4d75c9e
> 
> Description: Data Execution Prevention Violation
> Short Description: DEPViolation
> Exploitability Classification: EXPLOITABLE
> Recommended Bug Title: Exploitable - Data Execution Prevention Violation
> starting at <Unloaded_NPSWF32.dll>+0x135c9e (Hash=0x35022a3c.0x354a2a76)
> 
> User mode DEP access violations are exploitable.

The best way to address this may be to pull this one-off patch and the one-off fix we implemented in bug 132759 so that we don't special treat these events. Without pulling what we did in bug 132759 we'll be stuck trying to work up yet another one-off to address bug 493601.

Comment 5

[chris hofmann]

I looked at a sample of about 200 crash reports with this signature received over the last 14 days.  100% of the reports in this sample showed the module

NPSWF32.dll	10.0.32.18

which was released in the flash update on July 31, 2009

Comment 6

[chris hofmann]

charles,  any thoughts on this one.  we don't have a reproducible test case yet, but the ramp in the number of crash reports appears associated with the release of flash 10.0.32.18.  Its also possible the fix bug 493601 that went into Firefox 3.5.1 could have tickled additional problems.

Comment 7

[Tim Riley [:timr]]

That's http://apps.facebook.com/farmtown if others want to help try find a reproducible crash test case.

Comment 9

[Tim Riley [:timr]]

There are some reports of crashing firefox here on the Farmtown forum:
http://r1.slashkey.com/forum/showthread.php?t=90682

Comment 10

[Tim Riley [:timr]]

I have seen some reports of people fixing the crash by updating silverlight, upgrading Firefox and flash, and _reinstalling_ flash.

Some things people have done in farmtown that results in crashes are:
* zoom in and out
* get a job helping on another farm, then it crashes when you go that that farm
* having trees turned on (this is a strange world) and having lots of animals moving (see below)

----
From the SUMO page
http://support.mozilla.com/tiki-view_forum_thread.php?locale=lt&comments_parentId=389641&forumId=1

hey danielle here are a few thing i have done , to help keep me in farm-town first turn off trees uncheck box show tree's , i also freeze my animals so they dont walk around , i do recommend that if you need to change these settings to go to market place first, then make changes, then go back to your farm, if i try to make these changes while i am on my farm firefox crashes, and to problems with farmtown as of yesterday july 31, it seems to be a server problem 
----

I have not seen any of these myself yet.

Comment 11

[Johnny Stenback (:jst)]

Attachment: Stack from firefox debug build crashing.

This is a stack I got from a crash I saw after playing around on http://www.miniclip.com/games/monster-trucks-nitro/en/ for over 30 minutes. This does show us that the code introduced in bug 493601 is involved at in some of these crashes, but it's not clear whether those changes impact this crash in any way...

Comment 12

[Charles]

Hey guys. While I can reproduce a crash it none of them have matched the offsets yet.  This could be due to a OOM scenario.   Navigate around...zoom out...rinse...repeat...wait.  Took less than 5 minutes to crash.

http://apps.facebook.com/farmtown/?farm_id=1525788451&cid=unk:fsig&ref=unk:fsig&auth_token=d7857afe76e91e2480801cc5bc301a84

Comment 13

[cmtalbert]

http://crash-stats.mozilla.com/report/index/3beab13a-cc9c-40ad-8b5f-2d8a42090916?p=1

Ok repro'd it twice with these steps:

1. Start Farm Town
2. Go to a farm with a lot of stuff on it.
3. Turn on full screeen mode, set preferences to have "smooth scrolling", "follow character while walking" all the show preferences.
4. Click around to create a complicated walking path
5. Immediately go to preferences and UNCHECK "show flowers"

= Results = 
Crash on windows

= Other Notes =
* Doesn't crash on OS X.

I'm building debug now to see what I can get with a crash stack.  THis was done using build: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20090915 Minefield/3.7a1pre (.NET CLR 3.5.30729)

Comment 14

[cmtalbert]

Attachment: Stacks from the crash mentioned in comment 13

Vlad and I managed to capture this stack in the degbugger using the steps in comment 13.

A note on the steps - they are not 100%.  While attempting this we noticed that we'd only crash about 3 out of 5 attempts.

Comment 15

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

The reason we don't get breakpad (and thus no crash reports) for this particular case is that we're actually crashing -inside- breakpad, while trying to handle the crash, which is inside flash.

The actual crash happened inside flash, in a flash-started thread (non-main thread):

0df2f864 646e7c41 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
0df2fb88 646e824e NPSWF32+0x77c41
0df2fbb4 646f6005 NPSWF32+0x7824e
00000000 00000000 NPSWF32+0x86005

NPSWF32+0x77c41 is consistent with a run where we disabled breakpad, crash happened in the same spot.  So sounds like we might need adobe to figure out what's there.

Clint, can you put what version of flash you have installed here?  Also, I think we still have the minidump.. (flash.mdmp, inside your users temp dir?), which might be helpful for the adobe guys.

Comment 16

[Carsten Book [:Tomcat]]

also testcase (without using facebook apps) - load https://www.spokennetwork.com/editorial/Downloading_from_Spoken_Network.html

crashes after a few seconds (as example stack from a 3.0.14 build) :
http://crash-stats.mozilla.com/report/index/29590ce2-410f-4d60-be11-f7bc32090917

Comment 17

[Jim Mathies [:jimm]]

(In reply to comment #16)
> also testcase (without using facebook apps) - load
> https://www.spokennetwork.com/editorial/Downloading_from_Spoken_Network.html

Reproducible on 3.5.3 w/

    File: NPSWF32.dll
    Version: 10.0.32.18
    Shockwave Flash 10.0 r32

Also reproducible on trunk, although I haven't been able to break into a debug build yet due to lockups. Will keep playing with it.

Comment 18

[Charles]

(In reply to comment #2)
> the facebook farmtown app is frequently assocated with this particular
> signature, but that isn't the only crashing signature on farmtown.
> 
> top crashing signatures for farmtown in the last 14 days
> 9811 NPSWF32.dll@0x77bd0
> 4539 NPSWF32.dll@0x77540
> 1765 NPSWF32.dll@0xbc897
> 1450 NPSWF32.dll@0xbbff7
> 877 Flash Player@0xe4a5f
> 571 NPSWF32.dll@0x77c41
> 429 RealDefWindowProcWorker
> 
> and there are around 3341 uniq signatures related to farm town crashes.

Chris - can you confirm how you know there are 3341 unique signature related to farmtown?

Comment 19

[chris hofmann]

> Chris - can you confirm how you know there are 3341 unique signature 
> related to farmtown? 

looking at daily slices of the Socorro database I can look at all the crashing URLs reported, extract those where users reported being on facebook/farmtown app,  then count the unique signatures that collection of reports.

the url data isn't public due to privacy concerns, but we are able to report abastracts or sanitized information like this.

Comment 20

[trg16]

I hope this is pertinent to this bug... 100% of flash games on Kongregate.com crash the Branch build. I'm on a Mac and also had a Linux user confirm same behavior on Kongregate.com games. This has been commented in the Branch Thread Forum from the 17th and on (but I noticed these crashes a bit earlier)...

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2b1pre) Gecko/20090922 Namoroka/3.6b1pre

Comment 21

[chris hofmann]

it appears there is a lot of different problem surfacing on a lot of different games on kongregate.com.  The NPSWF32.dll@0x77bd0 is the top problem but here is a lost of the others for crashes received on 20090920

   5 NPSWF32.dll@0xb7b10 http://www.kongregate.com/games/ArmorGames/sonny-2

   2 NPSWF32.dll@0x2e83b7 http://www.kongregate.com/games/jamieyg3/sacred-seasons-mmorpg
   1 nsScriptLoader::StartLoad(nsScriptLoadRequest*, nsAString_internal const&) http://www.kongregate.com/games/PitchMobile/shield-defense
   1 nsIFrame::GetAncestorWithView() http://www.kongregate.com/
   1 nsHttpChannel::Connect(int) http://www.kongregate.com/games/JGames/pwong-2
   1 np32dsw.dll@0x20a2 http://www.kongregate.com/games/tmb_steve/merlins-revenge-1
   1 kernel32.dll@0x983e http://www.kongregate.com/games/garin/monsters-den-book-of-dread
   1 arena_chunk_init http://www.kongregate.com/games/Jiggmin/platform-racing-2
   1 _chkstk http://www.kongregate.com/games/raitendo/doeo?acomplete=doeo
   1 _cairo_surface_allocate_clip_serial http://www.kongregate.com/games/gameinabottle/gemcraft-chapter-0
   1 \N http://www.kongregate.com/games/JGames/pwong
   1 \N http://www.kongregate.com/games/DrunkenPorpoise/amaza-td
   1 SynthesizeFrame http://www.kongregate.com/games/Komix/seed-of-destruction
   1 RtlpCoalesceFreeBlocks http://www.kongregate.com/
   1 RtlEnterCriticalSection http://www.kongregate.com/games/inXile_Ent/super-stacker-2
   1 RtlActivateActivationContextUnsafeFast http://www.kongregate.com/games/weasel/thing-thing-arena-3
   1 PR_EnumerateAddrInfo http://www.kongregate.com/games/Void/mad-mutually-assured-destruction
   1 NPSWF32.dll@0xeaf4 http://www.kongregate.com/games/Foreverkul/toss-the-turtle
   1 NPSWF32.dll@0xddf0c http://www.kongregate.com/games/Jiggmin/platform-racing-2
   1 NPSWF32.dll@0xc200e http://www.kongregate.com/games/ArmorGames/sonny
   1 NPSWF32.dll@0xab06b http://www.kongregate.com/
   1 NPSWF32.dll@0xa97aa http://www.kongregate.com/games/JGames/pwong-2
   1 NPSWF32.dll@0xa0977 http://www.kongregate.com/games/tonypa/blockarelli
   1 NPSWF32.dll@0x9ce81 http://www.kongregate.com/games/PsychoGoldfish/generic-defense-game
   1 NPSWF32.dll@0x9c5e http://www.kongregate.com/games/JGames/pwong-2
   1 NPSWF32.dll@0x97cee http://www.kongregate.com/games/njellyfish/black-navy-war
   1 NPSWF32.dll@0x8489e http://www.kongregate.com/games/weasel/thing-thing-arena-3
   1 NPSWF32.dll@0x841db http://www.kongregate.com/games/ZigZaGame/the-great-war-of-prefectures
   1 NPSWF32.dll@0x84151 http://www.kongregate.com/games/Ivory/storm-the-house-3
   1 NPSWF32.dll@0x77c41 http://www.kongregate.com/games/GrubbyGames/incredibots-2
   1 NPSWF32.dll@0x4e92c http://www.kongregate.com/games/Fizzy/swords-and-sandals-3-solo-ultratus#game_tab_pane
   1 NPSWF32.dll@0x4830 http://www.kongregate.com/games/weasel/thing-thing-arena-3
   1 NPSWF32.dll@0x3ffd2 http://www.kongregate.com/games/weasel/thing-thing-arena-3
   1 NPSWF32.dll@0x3fcb3 http://www.kongregate.com/games/ArmorGames/sonny-2
   1 NPSWF32.dll@0x2e533b http://www.kongregate.com/games/kupo707/epic-battle-fantasy-2
   1 NPSWF32.dll@0x2ce8b7 http://www.kongregate.com/games/jamieyg3/sacred-seasons-mmorpg
   1 NPSWF32.dll@0x2ce8b7 http://www.kongregate.com/games/GrubbyGames/incredibots
   1 NPSWF32.dll@0x230221 http://www.kongregate.com/games/FlashkickGames/dig-defense
   1 NPSWF32.dll@0x220671 http://www.kongregate.com/games/dz2001/momentum-missile-mayhem-2
   1 NPSWF32.dll@0x2185e9 http://www.kongregate.com/games/Weasel/thing-thing-3?acomplete=thing+thing
   1 NPSWF32.dll@0x1604b6 http://www.kongregate.com/games/jamieyg3/sacred-seasons-mmorpg
   1 NPSWF32.dll@0x13f947 http://www.kongregate.com/games/Foreverkul/toss-the-turtle?acomplete=toss
   1 NPSWF32.dll@0x11ca6a http://www.kongregate.com/games/XGenStudios/stick-arena-ballistick
   1 GraphWalker::DoWalk(nsDeque&) http://www.kongregate.com/games/inXile_Ent/super-stacker-2
   1 GetGCThingFlags http://www.kongregate.com/games/struma/gateway-ii
   1 Flash Player@0xeea84 http://www.kongregate.com/games/kupo707/epic-battle-fantasy-2
   1 Flash Player@0xa46c3 http://www.kongregate.com/collabs/sounds/hugoleo777/a-good-day-to-die?ts=1250491430
   1 Flash Player@0x92160 http://www.kongregate.com/pages/pandemic-swine-flu-guide?gamereferral=Pandemic:%20American%20Swine
   1 Flash Player@0x9192b http://www.kongregate.com/games/gaby/zilch
   1 Flash Player@0x199b73 http://www.kongregate.com/games/GrubbyGames/incredibots-2?acomplete=incre
   1 CFBundleGetFunctionPointerForName http://www.kongregate.com/games/tmb_steve/merlins-revenge-3
   1 @0x0 http://www.kongregate.com/games/DJStatika/warlords-call-to-arms?acomplete=warlor

Comment 23

[RNicoletto]

(In reply to comment #22)
> *** Bug 521669 has been marked as a duplicate of this bug. ***

The crash report in that bug is a little bit different than the one in this bug: NPSWF32.dll@0xa10ff. Plus, all my crashes happened using FarmVille on Facebook (and NOT Farm Town). Actually there are over 2800 crash reports (and counting) around that signature ==> http://tinyurl.com/yg7afk3

Comment 24

[RNicoletto]

(In reply to comment #16)
> also testcase (without using facebook apps) - load
> https://www.spokennetwork.com/editorial/Downloading_from_Spoken_Network.html
> 
> crashes after a few seconds (as example stack from a 3.0.14 build) :
> http://crash-stats.mozilla.com/report/index/29590ce2-410f-4d60-be11-f7bc32090917

Confirmed ==> 1f14851f-85e7-4db6-8d04-2c8a92091011 

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3.

Comment 25

[chris hofmann]

volume on signature NPSWF32.dll@0x77bd0 seems to have dropped.  its running at 160-320 crashes per day for oct-nov 19.   maybe content side change have helped?

Comment 26

[Charles]

Recent testing on my end shows that changes on the Farmtown side has improved stability.  While we're injecting a fix for the previous related crashes I have not been able to reproduce the older crash reports with the same release player.

Comment 27

[Samuel Sidler (old account; do not CC)]

(In reply to comment #25)
> volume on signature NPSWF32.dll@0x77bd0 seems to have dropped.  its running at
> 160-320 crashes per day for oct-nov 19.   maybe content side change have
> helped?

Also keep in mind that the offsets of where Flash crashes change depending on the Firefox version, likely because of plug-in module changes.

Comment 28

[Charles]

Our release of Flash Player 10.0.42.34 should have fixed this issue.

Comment 29

[chris hofmann]

ok,  marking fixed.  I just scanned a bunch of reports from yesterday, but they are all NPSWF32.dll 10.0.32.18

if the signature has morfed as suggested in comment 27 then we will need a new bug filed or a way to figure out how to connect the new address.

Comment 30

[BMO Automation]

Version and milestone values are being reset to defaults as part of product refactoring.