561469 - Consistent but random crashes. with www.lordofultima.com loaded in a tab [@ RT40+0x01788abb]

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Description

[Ashu]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100423 Minefield/3.7a5pre (.NET CLR 3.5.30729) Firefox/3.6.3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100423 Minefield/3.7a5pre (.NET CLR 3.5.30729)

Since I started playing Lord of Ultima on www.lordofultima.com I've been seeing consistent crashes. I'm not sure if it is really related to the game itself because I've seen these crashes too while browsing on another tab.

Sometimes Breakpad catches the crash but it's always unable to submit the report because of some problem. Often the browser crashes without Breakpad catching it.

Every time it crashes though there is a big load on the hard drive.

Since it just happens out of the blue after running the browser for a while I cannot really provide any steps to reproduce it.

Reproducible: Always

Comment 1

[Tyler Downer [He/Him]]

https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report

Comment 2

[Ashu]

As described there I used WinDBg to get the stack trace. Unlike the problem described Minefield just hung. And it was hanging right away when it started. I hope this helps you in any way.

If you like I can give it another shot tomorrow.

Comment 3

[Ashu]

Attachment: WinDBg log file

Comment 4

[timeless]

hrm, that didn't work.

context:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
prompt/command:
1:020> g
- you did the right thing

context:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
prompt/command:
1:027> |* ~* kp

here you should have done 'g' as you did earlier. You may have to do this once for each plugin. the two lines preceding the '>' prompt should always be the same.

Our instructions predate the heavy use of OOPP. Sadly that means you're kind of a guinea pig as we work to improve them. Please work with us as we try to both improve our instructions and analyze your problem.

Comment 5

[Ashu]

I'll help you any way I can. I just need a step by step instruction set because I don't really know that software.

Btw I was able to rule out extensions as a possible source of trouble since it happened in safe mode too.

Comment 6

[timeless]

just follow the instructions from 

https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

as you did. however each time the view there gives you:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
?:???>

type 'g' <enter>

when you actually get to something which doesn't have that, then continue with '|* ~* kp' and the remaining commands.

Comment 7

[Ashu]

Attachment: New crash log

This log looks a lot more interesting. I hope it helps you in finding the issue.

Comment 8

[timeless]

yes, that's much better, thanks!

> Executable search path is: 
> ModLoad: 00400000 00406000   plugin-container.exe
...
> 1:031> g
...
> ModLoad: 01620000 01b9a000   C:\WINDOWS\system32\Macromed\Flash\NPSWF32.dll
...
> ModLoad: 78080000 78091000   C:\WINDOWS\system32\MSVCRT40.dll
...
> (eb4.11e0): Access violation - code c0000005 (first chance)
> eax=00000102 ebx=02a7c9dc ecx=7c802600 edx=7c91e514 esi=02a7c000 edi=02a7c9e0
> eip=017baaef esp=02c9ff28 ebp=02c9ffa0 iopl=0         nv up ei pl nz na po nc
> cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
> Missing image name, possible paged-out or corrupt data.
> Missing image name, possible paged-out or corrupt data.
> <Unloaded_RT40.dll>+0x17baaee:
> 017baaef ??              ???
...
> 1:045> |* !analyze -v -f
...
> FAULTING_IP: 
> RT40+1788abb
> 017baaef ??              ???

> EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
> .exr 0xffffffffffffffff
> ExceptionAddress: 017baaef (RT40+0x01788abb)
>    ExceptionCode: c0000005 (Access violation)
>   ExceptionFlags: 00000000
> NumberParameters: 2
>    Parameter[0]: 00000008
>    Parameter[1]: 017baaef
> Attempt to execute non-executable address 017baaef
> FAULTING_THREAD:  000011e0
> DEFAULT_BUCKET_ID:  BAD_INSTRUCTION_PTR
> PROCESS_NAME:  plugin-container.exe
> ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in "0x%08lx" verweist auf Speicher in "0x%08lx". Der Vorgang  "%s" konnte nicht auf dem Speicher durchgef hrt werden.
> EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in "0x%08lx" verweist auf Speicher in "0x%08lx". Der Vorgang  "%s" konnte nicht auf dem Speicher durchgef hrt werden.
> EXCEPTION_PARAMETER1:  00000008
> EXCEPTION_PARAMETER2:  017baaef
> WRITE_ADDRESS:  017baaef 
> FOLLOWUP_IP: 
> RT40+1788abb
> 017baaef ??              ???
> IP_MODULE_UNLOADED: 
> RT40+1788abb
> 017baaef ??              ???
> ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [ThreadStartAddress] from Frame:[0] on thread:[11e0]
> PRIMARY_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR
> BUGCHECK_STR:  APPLICATION_FAULT_BAD_INSTRUCTION_PTR_SOFTWARE_NX_FAULT
> LAST_CONTROL_TRANSFER:  from 017f597d to 017baaef
> STACK_TEXT:  
> WARNING: Stack unwind information not available. Following frames may be wrong.
> 02c9ff24 017f597d 000003e8 01b5b9e0 02a7c99c RT40+0x1788abb
> 02c9ffb4 7c80b729 02a7c99c 01b5b9e0 00000000 RT40+0x17c3949
> 02c9ffec 00000000 017bac36 02a7c99c 00000000 kernel32!BaseThreadStart+0x37

Roughly your crash is in an unloaded library, but there are a couple of problems:

1. we don't really know the name of the library
2. we don't know who's using it
3. we don't know why they unloaded it

http://www.cremarenco.com/i++/?p=9 suggests:

sxe ld MSVCRT40.dll
sxe ud MSVCRT40.dll

you'll do this before:
0:000> .childdbg 1

I guess...

when those breakpoints are hit, I *think* you should see:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
1:???> kp
...
1:???> g

the 'kp' and 'g' are things that you enter.

Since I think this is in the flash side, I'm going to move the bug their way.

Comment 9

[Ashu]

I am using Flash 10.1 RC2 just in case that information wasn't part of the log.

Comment 10

[timeless]

it wasn't, so thanks :)

-- it's something i need to improve in our instructions (i think i know how to do it, but i haven't had time).

Comment 11

[Ashu]

I reverted back to the latest release version of Flash 10.0 and the problem persists.

I'm wondering if it wouldn't be one of those things where OOPP and Flash just don't get along. Otherwise shouldn't OOPP be catching these crashes and keeping the main browser process running?

Comment 12

[Ashu]

Ok, I have confirmed that the crashes only occur if I have a tab with Lord of Ultima open. Without the tab it ran through the entire night without a crash. So it should be rather easy to reproduce. Go into the game (you can even play without registering I think) and let the tab sit there. After an undefinable amount of time but less than 2h as far as I can tell it will crash.

Comment 13

[timeless]

i'm marking this as new because i haven't seen any other reports which match this pattern and the reporter has provided enough windbg output for me to believe it isn't merely imagined (this comment/change is for bookkeeping purposes).

Comment 14

[Paul Betlem]

I spent some time testing this site and other than the use of Flash on the title page, the game itself does not use Flash. I suspect its a red herring. (If you uninstall Flash Player and go to the site, everything seems to work fine for me.) The game itself consumes a lot of memory so I'm wondering if its triggering a low-memory condition.

I tested for 8+ hours with periodic interaction using FF 3.6.3 withFP 10.1 RC2 and saw no problem.

Comment 15

[Ashu]

That could be but shouldn't Firefox's garbage collector prevent memory from running out?

Also I'm using Minefield so that might emphasize the problem.

Comment 16

[timeless]

paul: you probably need to use a 3.6.4candidate (roughly lorentz), a 3.6.5pre nightly (mozilla1.9.2 nightly, potentially a replacement 3.6.4candidate), or a 3.7pre nightly (mozilla-central). the crash involves out of process plugins which are a post 3.6.3 feature

Comment 17

[Ashu]

I think Paul might be on to something. Because of a huge memory leak that they introduced with a patch I've been forced to reload the tab all of the time to free the memory again. Since I've been doing this I haven't seen a crash any more. But when I let it hit about 1.55GB memory consumption (I have 2GB on this laptop in total) on the firefox.exe process it will cause the kind of crash that I described above.

The memory leak is obviously a bug or sloppy coding on behalf of the game but I'm not sure if Firefox should be crashing like that. I guess IPC for tabs will fix it eventually but it's certainly something that should be looked into.

Comment 18

[timeless]

aashish: so... the report you sent only shows the plugin host crashing. it doesn't show the browser crashing. if that's crashing, i need another report....

Comment 19

[Ashu]

Attachment: WinDbg Log 05-02

Here is another Log. 

I checked and when the memory usage on firefox.exe is really high the crash occurs. I am not sure as to what exactly happens. But memory running out seems to play a part somehow. It would be just too much of a coincident.

Comment 20

[Ashu]

Attachment: WinDbg Log with OOPP disabled 05-02

For good measurement I tried this again with OOPP disabled. Interestingly WinDbg said that the process was already exited (not by me) but it was still in the task manager. It was showing as locked up task. And it was still in the task bar but inaccessible.

Btw as a clarification I want to mention that in the previous crash log with OOPP enabled it looked pretty much the same but entering g moved it all along and led me to that Access Violation. But when that Access Violation is hit the firefox.exe vanishes from the Process List in the task manager. So if it really is the OOPP process that crashes then it still manages to kill firefox.exe right along.

Maybe these two logs together will help identify the core problem better. I hope I'm not confusing you with my comments.

Comment 21

[timeless]

Comment on attachment 442967 [details]
WinDbg Log with OOPP disabled 05-02

So, the reason this appears as an exit is because it really quit, because you really ran out of memory. This should be treated as a different problem. I've filed bug 563225 for it.

Comment 22

[timeless]

Comment on attachment 442961 [details]
WinDbg Log 05-02

>ModLoad: 71dd0000 71de5000   C:\WINDOWS\system32\msapsspc.dll
>ModLoad: 78080000 78091000   C:\WINDOWS\system32\MSVCRT40.dll

>IP_MODULE_UNLOADED: 
>sspc+11d735c
>011d735d ??              ???
>
>PRIMARY_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR
>
>BUGCHECK_STR:  APPLICATION_FAULT_BAD_INSTRUCTION_PTR_SOFTWARE_NX_FAULT
>
>LAST_CONTROL_TRANSFER:  from 76af54e3 to 011d735d
>
>STACK_TEXT:  
>WARNING: Frame IP not in any known module. Following frames may be wrong.
>02c3feb4 76af54e3 00000010 00000000 01835000 <Unloaded_sspc.dll>+0x11d735c
>02c3fedc 76b0adfe 011d735d 00000003 00000010 WINMM!DriverCallback+0x5c
>02c3ff18 76b0af02 00000010 40000060 b116ed08 WINMM!TimerCompletion+0xf4
>02c3ffb4 7c80b729 00000000 00150000 40000060 WINMM!timeThread+0x53
>02c3ffec 00000000 76b0aeaf 00000000 00000000 kernel32!BaseThreadStart+0x37

thanks. this time it looks like we lucked out and got more footprints of our culprit.

http://forums.techarena.in/operating-systems/1224290.htm seems to indicate this villain is not exactly harmless. According to this, it comes from frontpage. Does that sound familiar? If you could try renaming this library and then running firefox, i'd love to know if your problems go away.

http://xpdll.nirsoft.net/msapsspc_dll.html has vaguely helpful analysis of its internals (we could do better if i walked you through stuff, but i don't think we need to).

Comment 23

[Ashu]

Interesting. I don't actually have MS Office installed on this laptop though. I am using OpenOffice 3.2 instead. Maybe it's part of the .Net Framework too?

Either way I put old_ in front of the file name. We'll see if that is of any help.

Comment 24

[timeless]

msvc40 is *ancient*, so a library built against it would be ancient and shouldn't come from any modern software.

you can use procexp from http://sysinternals.com to ask it to "Verify" the library to confirm it's from Microsoft, and you can get properties on the file to read its version info....

the quick google search i did didn't indicate anyone else would use it. note that frontpage is generally somewhat independent of office.

Comment 25

[Ashu]

msapsspc.dll seems to be recreated every time I remove or rename it from system32. Also it doesn't seem to get loaded by firefox.exe or plugin-container.exe so I'm not sure how I should verify the dll. There doesn't seem to be a possibility in procexp to just check any dll you like.

The tooltip of the file says that it's from Microsoft and that it's a DPA-Client for Windows 32-Bit platforms. Also with a date of creation of 2004 it might have been part of Windows XP SP2.

I'm letting Antivir take a shot of checking for Viri but I don't think I caught one.

Comment 26

[timeless]

well... process explorer can be asked to verify all loaded libraries, and windbg shows that it's was loaded eventually.

there's probably a registry key we can use to block the library from loading into firefox/plugin-container temporarily for testing, but i'm too tired to look for that.

Comment 27

[Ashu]

Ok here is something to think about. When I play the game on IE8 it doesn't consume as much memory. Or it frees the memory up frequently so that the consumption stays stable. Maybe that javascript is just better optimized for IE8 or we actually have a problem in Minefield that causes it to leak memory on this site.

Comment 28

[Ashu]

I'm not even sure if anybody is still reading this bug but I wanted to add that Firefox 3.6.3 doesn't have this memory leak so it doesn't crash either.

Comment 29

[timeless]

since this isn't our code, and it isn't flashes, resolving as worksforme

Comment 30

[BMO Automation]

Version and milestone values are being reset to defaults as part of product refactoring.