Consistent but random crashes. with www.lordofultima.com loaded in a tab [@ RT40+0x01788abb]

RESOLVED WORKSFORME

Status

External Software Affecting Firefox
Flash (Adobe)
--
critical
RESOLVED WORKSFORME
8 years ago
2 years ago

People

(Reporter: Ashu, Unassigned)

Tracking

(Depends on: 1 bug, {crash})

Firefox Tracking Flags

(Not tracked)

Details

(crash signature)

Attachments

(2 attachments, 2 obsolete attachments)

40.10 KB, text/plain
Details
26.92 KB, text/plain
Details
(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100423 Minefield/3.7a5pre (.NET CLR 3.5.30729) Firefox/3.6.3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100423 Minefield/3.7a5pre (.NET CLR 3.5.30729)

Since I started playing Lord of Ultima on www.lordofultima.com I've been seeing consistent crashes. I'm not sure if it is really related to the game itself because I've seen these crashes too while browsing on another tab.

Sometimes Breakpad catches the crash but it's always unable to submit the report because of some problem. Often the browser crashes without Breakpad catching it.

Every time it crashes though there is a big load on the hard drive.

Since it just happens out of the blue after running the browser for a while I cannot really provide any steps to reproduce it.

Reproducible: Always
(Reporter)

Updated

8 years ago
Version: unspecified → Trunk
(Reporter)

Comment 2

8 years ago
As described there I used WinDBg to get the stack trace. Unlike the problem described Minefield just hung. And it was hanging right away when it started. I hope this helps you in any way.

If you like I can give it another shot tomorrow.
(Reporter)

Comment 3

8 years ago
Created attachment 441233 [details]
WinDBg log file
Attachment #441233 - Attachment mime type: application/octet-stream → text/plain
Component: General → IPC
Product: Firefox → Core
QA Contact: general → ipc

Comment 4

8 years ago
hrm, that didn't work.

context:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
prompt/command:
1:020> g
- you did the right thing

context:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
prompt/command:
1:027> |* ~* kp

here you should have done 'g' as you did earlier. You may have to do this once for each plugin. the two lines preceding the '>' prompt should always be the same.

Our instructions predate the heavy use of OOPP. Sadly that means you're kind of a guinea pig as we work to improve them. Please work with us as we try to both improve our instructions and analyze your problem.
(Reporter)

Comment 5

8 years ago
I'll help you any way I can. I just need a step by step instruction set because I don't really know that software.

Btw I was able to rule out extensions as a possible source of trouble since it happened in safe mode too.

Comment 6

8 years ago
just follow the instructions from 

https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

as you did. however each time the view there gives you:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
?:???>

type 'g' <enter>

when you actually get to something which doesn't have that, then continue with '|* ~* kp' and the remaining commands.
(Reporter)

Comment 7

8 years ago
Created attachment 441463 [details]
New crash log

This log looks a lot more interesting. I hope it helps you in finding the issue.
Attachment #441233 - Attachment is obsolete: true

Updated

8 years ago
Attachment #441463 - Attachment mime type: application/octet-stream → text/plain

Comment 8

8 years ago
yes, that's much better, thanks!

> Executable search path is: 
> ModLoad: 00400000 00406000   plugin-container.exe
...
> 1:031> g
...
> ModLoad: 01620000 01b9a000   C:\WINDOWS\system32\Macromed\Flash\NPSWF32.dll
...
> ModLoad: 78080000 78091000   C:\WINDOWS\system32\MSVCRT40.dll
...
> (eb4.11e0): Access violation - code c0000005 (first chance)
> eax=00000102 ebx=02a7c9dc ecx=7c802600 edx=7c91e514 esi=02a7c000 edi=02a7c9e0
> eip=017baaef esp=02c9ff28 ebp=02c9ffa0 iopl=0         nv up ei pl nz na po nc
> cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
> Missing image name, possible paged-out or corrupt data.
> Missing image name, possible paged-out or corrupt data.
> <Unloaded_RT40.dll>+0x17baaee:
> 017baaef ??              ???
...
> 1:045> |* !analyze -v -f
...
> FAULTING_IP: 
> RT40+1788abb
> 017baaef ??              ???

> EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
> .exr 0xffffffffffffffff
> ExceptionAddress: 017baaef (RT40+0x01788abb)
>    ExceptionCode: c0000005 (Access violation)
>   ExceptionFlags: 00000000
> NumberParameters: 2
>    Parameter[0]: 00000008
>    Parameter[1]: 017baaef
> Attempt to execute non-executable address 017baaef
> FAULTING_THREAD:  000011e0
> DEFAULT_BUCKET_ID:  BAD_INSTRUCTION_PTR
> PROCESS_NAME:  plugin-container.exe
> ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in "0x%08lx" verweist auf Speicher in "0x%08lx". Der Vorgang  "%s" konnte nicht auf dem Speicher durchgef hrt werden.
> EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in "0x%08lx" verweist auf Speicher in "0x%08lx". Der Vorgang  "%s" konnte nicht auf dem Speicher durchgef hrt werden.
> EXCEPTION_PARAMETER1:  00000008
> EXCEPTION_PARAMETER2:  017baaef
> WRITE_ADDRESS:  017baaef 
> FOLLOWUP_IP: 
> RT40+1788abb
> 017baaef ??              ???
> IP_MODULE_UNLOADED: 
> RT40+1788abb
> 017baaef ??              ???
> ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [ThreadStartAddress] from Frame:[0] on thread:[11e0]
> PRIMARY_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR
> BUGCHECK_STR:  APPLICATION_FAULT_BAD_INSTRUCTION_PTR_SOFTWARE_NX_FAULT
> LAST_CONTROL_TRANSFER:  from 017f597d to 017baaef
> STACK_TEXT:  
> WARNING: Stack unwind information not available. Following frames may be wrong.
> 02c9ff24 017f597d 000003e8 01b5b9e0 02a7c99c RT40+0x1788abb
> 02c9ffb4 7c80b729 02a7c99c 01b5b9e0 00000000 RT40+0x17c3949
> 02c9ffec 00000000 017bac36 02a7c99c 00000000 kernel32!BaseThreadStart+0x37

Roughly your crash is in an unloaded library, but there are a couple of problems:

1. we don't really know the name of the library
2. we don't know who's using it
3. we don't know why they unloaded it

http://www.cremarenco.com/i++/?p=9 suggests:

sxe ld MSVCRT40.dll
sxe ud MSVCRT40.dll

you'll do this before:
0:000> .childdbg 1

I guess...

when those breakpoints are hit, I *think* you should see:
ntdll!DbgBreakPoint:
7c91120e cc              int     3
1:???> kp
...
1:???> g

the 'kp' and 'g' are things that you enter.

Since I think this is in the flash side, I'm going to move the bug their way.
Component: IPC → Flash (Adobe)
Keywords: crash
Product: Core → Plugins
QA Contact: ipc → adobe-flash
Version: Trunk → unspecified
(Reporter)

Comment 9

8 years ago
I am using Flash 10.1 RC2 just in case that information wasn't part of the log.

Comment 10

8 years ago
it wasn't, so thanks :)

-- it's something i need to improve in our instructions (i think i know how to do it, but i haven't had time).
Version: unspecified → 10.x
(Reporter)

Comment 11

8 years ago
I reverted back to the latest release version of Flash 10.0 and the problem persists.

I'm wondering if it wouldn't be one of those things where OOPP and Flash just don't get along. Otherwise shouldn't OOPP be catching these crashes and keeping the main browser process running?
(Reporter)

Comment 12

8 years ago
Ok, I have confirmed that the crashes only occur if I have a tab with Lord of Ultima open. Without the tab it ran through the entire night without a crash. So it should be rather easy to reproduce. Go into the game (you can even play without registering I think) and let the tab sit there. After an undefinable amount of time but less than 2h as far as I can tell it will crash.

Comment 13

8 years ago
i'm marking this as new because i haven't seen any other reports which match this pattern and the reporter has provided enough windbg output for me to believe it isn't merely imagined (this comment/change is for bookkeeping purposes).
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Consistent but random crashes. Possibly www.lordofultima.com related. → Consistent but random crashes. with www.lordofultima.com loaded in a tab [@ RT40+0x01788abb]

Comment 14

8 years ago
I spent some time testing this site and other than the use of Flash on the title page, the game itself does not use Flash. I suspect its a red herring. (If you uninstall Flash Player and go to the site, everything seems to work fine for me.) The game itself consumes a lot of memory so I'm wondering if its triggering a low-memory condition.

I tested for 8+ hours with periodic interaction using FF 3.6.3 withFP 10.1 RC2 and saw no problem.
(Reporter)

Comment 15

8 years ago
That could be but shouldn't Firefox's garbage collector prevent memory from running out?

Also I'm using Minefield so that might emphasize the problem.

Comment 16

8 years ago
paul: you probably need to use a 3.6.4candidate (roughly lorentz), a 3.6.5pre nightly (mozilla1.9.2 nightly, potentially a replacement 3.6.4candidate), or a 3.7pre nightly (mozilla-central). the crash involves out of process plugins which are a post 3.6.3 feature
(Reporter)

Comment 17

8 years ago
I think Paul might be on to something. Because of a huge memory leak that they introduced with a patch I've been forced to reload the tab all of the time to free the memory again. Since I've been doing this I haven't seen a crash any more. But when I let it hit about 1.55GB memory consumption (I have 2GB on this laptop in total) on the firefox.exe process it will cause the kind of crash that I described above.

The memory leak is obviously a bug or sloppy coding on behalf of the game but I'm not sure if Firefox should be crashing like that. I guess IPC for tabs will fix it eventually but it's certainly something that should be looked into.

Comment 18

8 years ago
aashish: so... the report you sent only shows the plugin host crashing. it doesn't show the browser crashing. if that's crashing, i need another report....
(Reporter)

Comment 19

8 years ago
Created attachment 442961 [details]
WinDbg Log 05-02

Here is another Log. 

I checked and when the memory usage on firefox.exe is really high the crash occurs. I am not sure as to what exactly happens. But memory running out seems to play a part somehow. It would be just too much of a coincident.
(Reporter)

Comment 20

8 years ago
Created attachment 442967 [details]
WinDbg Log with OOPP disabled 05-02

For good measurement I tried this again with OOPP disabled. Interestingly WinDbg said that the process was already exited (not by me) but it was still in the task manager. It was showing as locked up task. And it was still in the task bar but inaccessible.

Btw as a clarification I want to mention that in the previous crash log with OOPP enabled it looked pretty much the same but entering g moved it all along and led me to that Access Violation. But when that Access Violation is hit the firefox.exe vanishes from the Process List in the task manager. So if it really is the OOPP process that crashes then it still manages to kill firefox.exe right along.

Maybe these two logs together will help identify the core problem better. I hope I'm not confusing you with my comments.

Updated

8 years ago
Attachment #442961 - Attachment mime type: application/octet-stream → text/plain

Updated

8 years ago
Attachment #442967 - Attachment mime type: application/octet-stream → text/plain

Updated

8 years ago
Depends on: 563225

Comment 21

8 years ago
Comment on attachment 442967 [details]
WinDbg Log with OOPP disabled 05-02

So, the reason this appears as an exit is because it really quit, because you really ran out of memory. This should be treated as a different problem. I've filed bug 563225 for it.
Attachment #442967 - Attachment is obsolete: true

Comment 22

8 years ago
Comment on attachment 442961 [details]
WinDbg Log 05-02

>ModLoad: 71dd0000 71de5000   C:\WINDOWS\system32\msapsspc.dll
>ModLoad: 78080000 78091000   C:\WINDOWS\system32\MSVCRT40.dll

>IP_MODULE_UNLOADED: 
>sspc+11d735c
>011d735d ??              ???
>
>PRIMARY_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR
>
>BUGCHECK_STR:  APPLICATION_FAULT_BAD_INSTRUCTION_PTR_SOFTWARE_NX_FAULT
>
>LAST_CONTROL_TRANSFER:  from 76af54e3 to 011d735d
>
>STACK_TEXT:  
>WARNING: Frame IP not in any known module. Following frames may be wrong.
>02c3feb4 76af54e3 00000010 00000000 01835000 <Unloaded_sspc.dll>+0x11d735c
>02c3fedc 76b0adfe 011d735d 00000003 00000010 WINMM!DriverCallback+0x5c
>02c3ff18 76b0af02 00000010 40000060 b116ed08 WINMM!TimerCompletion+0xf4
>02c3ffb4 7c80b729 00000000 00150000 40000060 WINMM!timeThread+0x53
>02c3ffec 00000000 76b0aeaf 00000000 00000000 kernel32!BaseThreadStart+0x37

thanks. this time it looks like we lucked out and got more footprints of our culprit.

http://forums.techarena.in/operating-systems/1224290.htm seems to indicate this villain is not exactly harmless. According to this, it comes from frontpage. Does that sound familiar? If you could try renaming this library and then running firefox, i'd love to know if your problems go away.

http://xpdll.nirsoft.net/msapsspc_dll.html has vaguely helpful analysis of its internals (we could do better if i walked you through stuff, but i don't think we need to).
(Reporter)

Comment 23

8 years ago
Interesting. I don't actually have MS Office installed on this laptop though. I am using OpenOffice 3.2 instead. Maybe it's part of the .Net Framework too?

Either way I put old_ in front of the file name. We'll see if that is of any help.

Comment 24

8 years ago
msvc40 is *ancient*, so a library built against it would be ancient and shouldn't come from any modern software.

you can use procexp from http://sysinternals.com to ask it to "Verify" the library to confirm it's from Microsoft, and you can get properties on the file to read its version info....

the quick google search i did didn't indicate anyone else would use it. note that frontpage is generally somewhat independent of office.
(Reporter)

Comment 25

8 years ago
msapsspc.dll seems to be recreated every time I remove or rename it from system32. Also it doesn't seem to get loaded by firefox.exe or plugin-container.exe so I'm not sure how I should verify the dll. There doesn't seem to be a possibility in procexp to just check any dll you like.

The tooltip of the file says that it's from Microsoft and that it's a DPA-Client for Windows 32-Bit platforms. Also with a date of creation of 2004 it might have been part of Windows XP SP2.

I'm letting Antivir take a shot of checking for Viri but I don't think I caught one.

Comment 26

8 years ago
well... process explorer can be asked to verify all loaded libraries, and windbg shows that it's was loaded eventually.

there's probably a registry key we can use to block the library from loading into firefox/plugin-container temporarily for testing, but i'm too tired to look for that.
(Reporter)

Comment 27

8 years ago
Ok here is something to think about. When I play the game on IE8 it doesn't consume as much memory. Or it frees the memory up frequently so that the consumption stays stable. Maybe that javascript is just better optimized for IE8 or we actually have a problem in Minefield that causes it to leak memory on this site.
(Reporter)

Comment 28

8 years ago
I'm not even sure if anybody is still reading this bug but I wanted to add that Firefox 3.6.3 doesn't have this memory leak so it doesn't crash either.

Comment 29

8 years ago
since this isn't our code, and it isn't flashes, resolving as worksforme
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME
(Assignee)

Updated

7 years ago
Crash Signature: [@ RT40+0x01788abb]

Comment 30

2 years ago
Version and milestone values are being reset to defaults as part of product refactoring.
Version: 10.x → unspecified
You need to log in before you can comment on or make changes to this bug.