429713 - playing .MOV in quicktime 7.1.6 consistently crashes [@ free - _releaseobject - ns4xPluginInstance::GetJSObject]

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect)

Description

[martin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9pre) Gecko/2008041606 Minefield/3.0pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9pre) Gecko/2008041606 Minefield/3.0pre

1. go to http://saturn.jpl.nasa.gov/multimedia/videos/movies/PIA08117.mov
2. file will almost never play until end
3. if you do other stuff meanwhile it will crash faster, my guess is that QT is overwriting random parts of firefox address space

Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[martin]

My browser also crashes on pretty much all other .MOV files after 10-90 seconds of playback. I've had this problem for several weeks atleast using minefield.

Comment 2

[martin]

I've submitted at least two reports using the new "crash reporting" for this bug, look for someone who used contact e-mail "[e-mail]" and mentions ".MOV" in the "details" textbox.

Comment 3

[martin]

This bug also does NOT repro if the .MOV file is in cache. 

If this bug seems hard to repro, try these steps.. use "Tools :: Clear Private Data" and check only the "Cache" checkbox then press okay.

Also, it seems easier if you restart the .MOV a couple of times by putting down the cursor in the address bar and pressing ENTER (that way the .MOV restarts and this will in case cases cause a direct crash).

Comment 4

[martin]

I just looked in about:crashes and I had several of the .MOV crashes recorded in there. All of these crash reports are related to the bug I'm describing above:

http://crash-stats.mozilla.com/report/index/9b92e628-0d77-11dd-aa58-0013211cbf8a
http://crash-stats.mozilla.com/report/index/c7e1c897-0d77-11dd-9aa8-0013211cbf8a
http://crash-stats.mozilla.com/report/index/656da000-0d7b-11dd-807a-0013211cbf8a
http://crash-stats.mozilla.com/report/pending/c10d963d-0d7c-11dd-9b83-001cc45a2ce4

Comment 5

[martin]

Note 1: I never see any crashes in FF3 using the exact same machine, same QuickTime version and same .MOV files.

Note 2: it's not possible to download the latest QuickTime for win2k because they only support XP and Vista these days. Maybe all win2k users are upgraded to 7.1.6 and then no further? I'm not sure.

I also tried to install this security update for 7.1.6 (available here; http://www.apple.com/downloads/macosx/apple/security_updates/securityupdatequicktime716forwindows.html ) but I can still repro the crash. This crash report was submitted after installing the security update:
http://crash-stats.mozilla.com/report/index/0eb6708c-0d7f-11dd-938d-001321b13766

Comment 6

[Matthias Versen [:Matti]]

0  	mozcrt19.dll  	free  	 jemalloc.c:6035
1 	xul.dll 	_releaseobject 	mozilla/modules/plugin/base/src/ns4xPlugin.cpp:1731
2 	xul.dll 	ns4xPluginInstance::GetJSObject 	mozilla/modules/plugin/base/src/ns4xPluginInstance.cpp:1488
3 	xul.dll 	nsHTMLPluginObjElementSH::GetPluginJSObject 	mozilla/dom/src/base/nsDOMClassInfo.cpp:9321
4 	xul.dll 	nsHTMLPluginObjElementSH::SetupProtoChain 	mozilla/dom/src/base/nsDOMClassInfo.cpp:8903
5 	xul.dll 	nsHTMLPluginObjElementSH::PostCreate 	mozilla/dom/src/base/nsDOMClassInfo.cpp:9013
6 	xul.dll 	XPCWrappedNative::GetNewOrUsed 	mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:546
7 	xul.dll 	XPCWrappedNative::GetNewOrUsed 	mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:416
8 	xul.dll 	XPCConvert::NativeInterface2JSObject 	mozilla/js/src/xpconnect/src/xpcconvert.cpp:1106
9 	xul.dll 	XPCConvert::NativeData2JS 	mozilla/js/src/xpconnect/src/xpcconvert.cpp:481
10 	xul.dll 	XPCWrappedNative::CallMethod 	mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2456

Comment 7

[martin]

It seems I have several npqtplugin*.dll files on this system:

C:\Program Files>dir /b/s npqtplugin3.dll
C:\Program Files\Internet Explorer\PLUGINS\npqtplugin3.dll
C:\Program Files\Mozilla Firefox\plugins\npqtplugin3.dll
C:\Program Files\mozilla.org\SeaMonkey\plugins\npqtplugin3.dll
C:\Program Files\QuickTime\Plugins\npqtplugin3.dll

C:\Program Files>dir Minefield\firefox.exe
 Volume in drive C is SYSTEM
 Volume Serial Number is 24C7-80C2

 Directory of C:\Program Files\Minefield

2008-04-19  12:55               80,384 firefox.exe
               1 File(s)         80,384 bytes
               0 Dir(s)     453,976,064 bytes free

C:\Program Files>dir Minefield\plugins\*
 Volume in drive C is SYSTEM
 Volume Serial Number is 24C7-80C2

 Directory of C:\Program Files\Minefield\plugins

2008-01-27  17:54       <DIR>          .
2008-01-27  17:54       <DIR>          ..
2008-04-19  12:55               59,904 npnul32.dll
               1 File(s)         59,904 bytes
               2 Dir(s)     453,976,064 bytes free

Comment 8

[martin]

WHen I look at firefox.exe in "process explorer" I was that:

FF3 minefield uses:
C:\Program Files\QuickTime\Plugins\npqtplugin.dll
FF2 uses:
C:\Program Files\Mozilla Firefox\plugins\npqtplugin.dll

but acoording to windiff these files are identical, so I believe that something changed in FF3 minefield which causes the QT crashes.

Comment 9

[timeless]

blacklist time.

reporter: please list all plugins from about:plugins.
for your own reference, load about:config, toggle plugin.expose_full_path
load about:plugins again, it'll tell you the paths it's actually using for your plugins

see also: 
http://kb.mozillazine.org/Plugin_scanning
plugin.scan.Quicktime 100
plugin.scan.plid.all false

please try renaming all the quicktime plugins to *.dul (from *.dll). if the crash goes away, please indicate that.

Comment 10

[martin]

First, thanks for the links timeless, I was actually trying to find information like this yesterday but you guys got so much stuff all over the place so it's hard to find what you need sometimes :-)

Second, I think it would be a good idea to try to find some other user with w2k and qt 7.1.6 just to assert that this is not an issue on my machine only (which I really don't think it is but still). Or is it possible to query the crash data to see if such crashes are being reported automatically by other users.

The bug is still reproducible on minefield from 21st of april (surprise surprise!), this time I got a 41 frames deep jemalloc stack (same as before sort of, still probably stack corruption). Details here:
http://crash-stats.mozilla.com/report/index/92acf475-0fe1-11dd-85da-001cc4e2bf68

Restarting minefield with "plugin.scan.Quicktime 100" yields no crash because instead of QT plugin running I get the usual boring open/save dialog. I don't think "plid.false" is interestin either because the only PLID entry I got in regedit is for flash.

If I rename "C:\Program Files\QuickTime\Plugins\npqtplugin.dll" to "C:\Program Files\QuickTime\Plugins\npqtplugin.dul" and restart minefield then the plugin is not loaded and I also get the open/save dialog instead of playback (this is with the about:config stuff reverted to defaults again ofc). Why is it interesting to try with *.dul btw?

Comment 11

[martin]

Attachment: copy paste from about:plugins on the relevant machine showing version numbers

Comment 12

[martin]

If you got some ninja windbg commands, I'd be happy to run those (just specify the commands in great detail because I'm kind of new to windbg). I'm thinking for instance that it might be possible to debug a stack corruption by putting a data access break point on the memory address that holds the pointer which is passed to jemalloc's free()? I've no idea how to do that but it sounds like a good idea. Also, if that can't work I'm interested in know why :)

I mean, suppose the stack looks like this:

STACK_ITEM_A
STACK_ITEM_B
STACK_BUFFER_C
STACK_ITEM_D
STACK_ITEM_E

and D is a pointer allocated with jemalloc. Now, suppose that some other buggy closed source thread tries to write to STACK_BUFFER_C (which might be video data coming into some cairo graphics buffer or whatever, I'm just speculating here). Then that write to STACK_BUFFER_C might overrun and hit STACK_ITEM_D which we don't notice at all (at first). But, as it turns out we didn't need STACK_ITEM_D anyway so we're not using it (again, no errors because we're not using it) but at some point STACK_ITEM_D will be free'd by jemalloc. At that point STACK_ITEM_D is a random pointer and of course jemalloc chokes on it.

Now, by putting a data access breakpoint on STACK_ITEM_D, couldn't we find the culprit thread (and function) which is overwriting STACK_ITEM_D?

The part I don't understand about how to accomplish this is; A) how can we know what STACK_ITEM_D will be? i.e. How can we know the ptr that will cause jemalloc/free to choke later on?

Why do I even want to find the function that performs the overrun? Well, I think it might be interesting to put a breakpoint on that function when running inside FF2 with QT 7.1.6 because then we can figure out why this bug happens in FF3 but not in FF2.

Comment 13

[timeless]

dul is just a convenience. we only load files named np*.dll on windows, so any other name works. but you can easily rename between dll and dul and still recognize that it was a dll w/o getting confused.

oh right. you're volunteering. perfect.

i'm doing this from memory, so please try to experiment and ask on irc.mozilla.org for help if you get stuck.

here's your starting point:
.logopen /t c:\ff-qt-429713
bp xul!_memalloc "kp 10; g"
bp xul!_memfree "kp 10; g"
bp xul!_createobject "kp 10; g"
bp xul!_retainobject "kp 10; g"
bp xul!_releaseobject "kp 10; g"
bp xul!_construct "kp 10; g"
bp xul!_releasevariantvalue "kp 10; g"

run this before you load the quicktime page/file.
there's also
.logclose

if you need help, .hh .hh or whatever

when you're done, attach the log file (please use .zip)

if things get really bad, we'll probably need to figure out which allocator they're using (which is fairly likely).

for that you're going to need to try other strange things,

one would be:
bp npqtplugin3!malloc "kp 5; g"
bp npqtplugin3!calloc "kp 5; g"
bp npqtplugin3!realloc "kp 5; g"
bp npqtplugin3!free "kp 5; g"

if that specific item doesn't work, 
http://www.mozilla.org/quality/help/dependency-walker.html
run firefox under that, and then browse to npqtplugin3 in the tree, and expand the tree, you're looking for msvcr*, whichever one it is, e.g. msvcr71.dll
do:
bp msvcr71!malloc "kp 5; g"
(plus the other entrypoints as listed above)

basically these bugs are because instead of using the public api we expose, they're using their own private allocator and giving us the data assuming we use the same underlying allocator (which we aren't required to do, and as of b5 do not do).

the goal of these traces is to figure out which pointer is allocated by whom and wrongly passed to another module. this becomes the evidence/smoking gun showing that they're broken (we know this already but...), and it also should help them fix their code because it's the specific path that they need to fix to use our allocator.

Comment 14

[martin]

This is really cool, I'm happy to help! I'll try to do one thing at a time here. The first thing I did was the 7 breakpoint logging thing:

1. start minefield 23rd apr in windbg, load symbols, activate logging, set xul "kp 10;g"-breakpoints and then after "g":
A) point the browser to a .MOV file (whenever windbg grabs attention I just "g" it)
B) wait 20 secs with movie playing
C) put the cursor in the address field
D) press ENTER to reload the .MOV file

At this point I start getting these really weird "invalid locking sequence" exceptions popping up in windbg. I try to just "g" them at both first and second chance but they just keep on coming (I saw at least 20 of them). This is what they typically look like in windbg:

(994.8f4): Invalid lock sequence - code c000001e (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=03f001f0 ebx=0012f88c ecx=00000004 edx=0000001c esi=03f01fc0 edi=021bf3a0
eip=03f001f0 esp=0012f820 ebp=00000000 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00010206
03f001f0 f001f0          lock add eax,esi


(994.8f4): Invalid lock sequence - code c000001e (!!! second chance !!!)
eax=03f001f0 ebx=0012f88c ecx=00000004 edx=0000001c esi=03f01fc0 edi=021bf3a0
eip=03f001f0 esp=0012f820 ebp=00000000 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00000206
03f001f0 f001f0          lock add eax,esi

Please note that this is very different from the real crash. I actually tried to re-run the repro steps without having the "kp 10;g"-breakpoints activated and just like I thought the real crash is an access violation: 

(e10.a78): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=036701f0 ebx=0012f88c ecx=00000004 edx=0000001c esi=03671fc0 edi=01ccd3a0
eip=03670200 esp=0012f820 ebp=00000000 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00010206
03670200 0002            add     byte ptr [edx],al          ds:0023:0000001c=??

Despite this attempt being less than stellar, I'm attaching the log (which has the whole story in it).

QUESTION #1: Since I'm actually able to "g" these exceptions, does that mean that they are caught and ignored in firefox? Or does a "second chance" exception _always_ mean that a crash would have happened had the debugger not been attached?

QUESTION #2: I think I need some kind of way to automatically "g" these exceptions. How can I do that?

Comment 15

[martin]

Attachment: log from the debugging session I mention in comment #14

Comment 16

[martin]

Attachment: this is another run doing the steps described in comment 14 (this time no locking sequence exceptions)

Comment 17

[martin]

I tried to redo the steps to get a stack for the "invalid locking sequence" exceptions but that time it didnt get any exceptions. maybe it was a one off thing due to the particular stuff that was overwritten in memory during that run. During the second run (attacment id=317353) things went sort of like expected and I got the usual type of access violation.

Comment 18

[timeless]

http://developer.mozilla.org/en/docs/Using_the_Mozilla_source_server

I haven't gotten around to using this, but it's probably worth it.

first, sadly _createobject will need some smarter breakpoints

1702     npobj = aClass->allocate(npp, aClass);
1704     npobj = (NPObject *)PR_Malloc(sizeof(NPObject));

for these two lines, you cna start by using f9 to set basic breakpoints.

bl

will list those breakpoints. then use bp .... "kp 1; g" to change them to give you one line of stack (i want to know which path they took).

1715   return npobj;

again, use f9 to get a basic breakpoint

bp ... "dt npobj; g" to change the breakpoint to tell us the pointer it was returning.

I think you might need to use the malloc tracing stuff. but before we do that...

bp xul!NPObjWrapperPluginDestroyedCallback

with the source server, and maybe step through a bit (switch to asm mode from source mode so that step into does something useful)

anyway, that log doesn't show quite the same death, and the only retain/releases are balanced and seem to be gecko internals.

fwiw since you said the problem relates to cache/noncache, you can use the techniques here to log the npapi methods that deal w/ retrieving web content :)

Comment 19

[martin]

While playing a mp3-file and opening the plugin-settings dialog I got another Quicktime crash with a pretty interesting stack. I saved a full memory dump using ".dump /ma blah" in windbg in case someone else wants to have a look:

http://files.minimum.se/bug_attachments/ff3/bug_429713_quicktime_consistently_crashes_on_win2k/qt_mp3_play_plugin_options_twice_dump_ma.zip

This dump was recorded by starting minefield 28th april using "windbg firefox.exe".

Basically, it's an AV in RtlAllocateHeap which can happen if:
A) bug in RtlAllocateHeap (very unlikely)
B) maybe the code for RtlAllocateHeap was overwritten with random code so that it's executing some crazy messed up thing (I don't think this is very likely, but I'm not sure)
C) a bad heap handle has passed into RtlAllocateHeap (this might very well be what just happened if the stack was overwritten, and we're definately dealing with some kind of randomly overwritten memory here).

Comment 20

[timeless]

typically those mean that they allocated memory using their allocator (typically msvcr*) and passed it via an NPAPI method with a contract that said they were supposed to use the NPAPI allocator.

Hence the requests for malloc/calloc breakpoints.

Comment 21

[martin]

I can repro this bug in minefield from 2007060104 but it's _a lot_ harder to get the bug to trigger on that build. I did it once with minefield started from windbg and then it was one of those lock sequence exceptions again. Then I did it with a 1st june 2007 minefield running normally, it crashed and the crash reporter utility came up, it tried to submit the crash report but then it just said "failed to submit report". I might try some other builds as well if I find the time for it.

Comment 22

[martin]

Attachment: kp5's *alloc from msvcrt and !analyze with src-server attached

These stacks didn't look very interesting to me, attaching anyway. I also looked inside the npqtplugin module with the "X" command but I didn't find any malloc etc in there. I will try the steps in comment #18 later.

Comment 23

[timeless]

this sure doesn't have the evidence i was hoping for :(

can you run these:
0:014> lm m msvc*
0:014> x *!*alloc*

again after you crash. afaiu the msvcr that ships w/ windows is 6, and the one we were expected to use was 7.1. but... we've been using 8 for a while, so i can't think of why the jemalloc switch would have been any more or less harmful than the  fact that we're not using 7.1's allocator anymore :(

we definitely need to figure out the ownership involved...

Comment 24

[martin]

I got another clue for thus bug. When I run it under full pageheap, the AV becomes 100% consistent, it's always located in the function xul!nsNPObjWrapper::GetNewOrUsed. It seems that npobj is pointing to a commited chunk of virtual memory which is marked with the PAGE_NOACCESS flag.

(1174.9a8): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=0012ecfc ecx=60a0f524 edx=607ea604 esi=04eb6ff0 edi=00000000
eip=60b28149 esp=0012ec0c ebp=0012ec30 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000             efl=00010206
xul!nsNPObjWrapper::GetNewOrUsed+0x1d:
60b28149 813eeca9d460    cmp     dword ptr [esi],offset xul!nsJSObjWrapper::sJSObjWrapperNPClass (60d4a9ec) ds:0023:04eb6ff0=????????
0:000> kpn 200
 # ChildEBP RetAddr  
00 0012ec30 60b2ddc4 xul!nsNPObjWrapper::GetNewOrUsed(struct _NPP * npp = 0x02c23ad8, struct JSContext * cx = 0x028157c0, struct NPObject * npobj = 0x04eb6ff0)+0x1d [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\modules\plugin\base\src\nsjsnpruntime.cpp @ 1677]
01 0012ec50 60a7a225 xul!ns4xPluginInstance::GetJSObject(struct JSContext * cx = 0x028157c0)+0x34 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\modules\plugin\base\src\ns4xplugininstance.cpp @ 1488]
02 0012ecb8 60a8c1fa xul!nsHTMLPluginObjElementSH::GetPluginJSObject(struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x0282d0a0, class nsIPluginInstance * plugin_inst = 0x02c23ac0, struct JSObject ** plugin_obj = 0x0012ecfc, struct JSObject ** plugin_proto = 0x0012ecf8)+0x57 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dom\src\base\nsdomclassinfo.cpp @ 9356]
03 0012ed04 60adc92c xul!nsHTMLPluginObjElementSH::SetupProtoChain(class nsIXPConnectWrappedNative * wrapper = 0x02862a60, struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x0282d0a0)+0x91 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dom\src\base\nsdomclassinfo.cpp @ 8929]
04 0012ed18 604e6f65 xul!nsHTMLPluginObjElementSH::PostCreate(class nsIXPConnectWrappedNative * wrapper = 0x02862a60, struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x0282d0a0)+0x2f [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dom\src\base\nsdomclassinfo.cpp @ 9039]
05 0012ee60 604e6f8e xul!XPCWrappedNative::GetNewOrUsed(class XPCCallContext * ccx = <Memory access error>, class nsISupports * Object = <Memory access error>, class XPCWrappedNativeScope * Scope = <Memory access error>, class XPCNativeInterface * Interface = <Memory access error>, int isGlobal = <Memory access error>, class XPCWrappedNative ** resultWrapper = <Memory access error>)+0x1095 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcwrappednative.cpp @ 546]
06 0012efb0 604ebfbe xul!XPCWrappedNative::GetNewOrUsed(class XPCCallContext * ccx = <Memory access error>, class nsISupports * Object = <Memory access error>, class XPCWrappedNativeScope * Scope = <Memory access error>, class XPCNativeInterface * Interface = <Memory access error>, int isGlobal = <Memory access error>, class XPCWrappedNative ** resultWrapper = <Memory access error>)+0x10be [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcwrappednative.cpp @ 416]
07 0012f018 604eb978 xul!XPCConvert::NativeInterface2JSObject(class XPCCallContext * ccx = 0x0282a24c, class nsIXPConnectJSObjectHolder ** dest = 0x0282a244, class nsISupports * src = 0x0282a2b4, struct nsID * iid = 0x0282a2bc, struct JSObject * scope = 0x0282a2c4, int allowNativeWrapper = 42115788, int isGlobal = 42115796, unsigned int * pErr = 0x0282a2dc)+0x13e [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcconvert.cpp @ 1107]
08 0012f094 604e9818 xul!XPCConvert::NativeData2JS(class XPCCallContext * ccx = 0x0012f2a8, long * d = 0x0012f0c8, void * s = 0x0012f150, class nsXPTType * type = 0x0012f0c6, struct nsID * iid = 0x0012f1d0, struct JSObject * scope = 0x03d18c60, unsigned int * pErr = 0x0012f0f8)+0xa8 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcconvert.cpp @ 481]
09 0012f284 604ef000 xul!XPCWrappedNative::CallMethod(class XPCCallContext * ccx = 0x60a0f524, XPCWrappedNative::CallMode mode = CALL_GETTER (1))+0x5e8 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcwrappednative.cpp @ 2475]
0a 0012f34c 6010b27b xul!XPC_WN_GetterSetter(struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x03d18c60, unsigned int argc = 0, long * argv = 0x02c9604c, long * vp = 0x0012f3d4)+0x150 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\xpconnect\src\xpcwrappednativejsops.cpp @ 1505]
0b 0012f418 6010a9e7 js3250!js_Invoke(struct JSContext * cx = 0x028157c0, unsigned int argc = 0, long * vp = 0x02c96044, unsigned int flags = 2)+0x2bb [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsinterp.c @ 1296]
0c 0012f44c 6011614f js3250!js_InternalInvoke(struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x03d18c60, long fval = 74559008, unsigned int flags = 0, unsigned int argc = 0, long * argv = 0x00000000, long * rval = 0x0012f4f8)+0xe7 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsinterp.c @ 1368]
0d 0012f4ac 60117a93 js3250!js_GetPropertyHelper(struct JSContext * cx = 0x04eb6ff0, struct JSObject * obj = 0x038c99a0, long id = 0, long * vp = 0x03d18c40, struct JSPropCacheEntry ** entryp = 0x601a48e1)+0x1cf [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsobj.c @ 3711]
0e 0012f62c 6010b33e js3250!js_Interpret(struct JSContext * cx = 0x028157c0)+0xa53 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsinterp.c @ 4175]
0f 0012f6ec 60105709 js3250!js_Invoke(struct JSContext * cx = 0x028157c0, unsigned int argc = 1, long * vp = 0x02c96024, unsigned int flags = 0)+0x37e [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsinterp.c @ 1313]
10 0012f728 60554302 js3250!JS_CallFunctionValue(struct JSContext * cx = 0x028157c0, struct JSObject * obj = 0x03d3b400, long fval = 66369184, unsigned int argc = 1, long * argv = 0x02c96020, long * rval = 0x0012f760)+0xb9 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\js\src\jsapi.c @ 5054]
11 0012f794 60568995 xul!nsJSContext::CallEventHandler(class nsISupports * aTarget = 0x03cb1c70, void * aScope = 0x038c3b60, void * aHandler = 0x03f4b6a0, class nsIArray * aargv = 0x028113b0, class nsIVariant ** arv = 0x0012f7dc)+0x192 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dom\src\base\nsjsenvironment.cpp @ 1962]
12 0012f8b8 605641a7 xul!nsJSEventListener::HandleEvent(class nsIDOMEvent * aEvent = 0x0280af30)+0x20d [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dom\src\events\nsjseventlistener.cpp @ 250]
13 0012f9c0 6050c71f xul!nsEventListenerManager::HandleEventSubType(struct nsListenerStruct * aListenerStruct = 0x607ea604, class nsIDOMEventListener * aListener = 0x03cb0ae0, class nsIDOMEvent * aDOMEvent = 0x0280af30, class nsISupports * aCurrentTarget = 0x60a0f524, unsigned int aPhaseFlags = 0x12fa14)+0x79 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\content\events\src\nseventlistenermanager.cpp @ 1080]
14 0012fa14 605020ab xul!nsEventListenerManager::HandleEvent(class nsPresContext * aPresContext = 0x00636300, class nsEvent * aEvent = 0x0012fb38, class nsIDOMEvent ** aDOMEvent = 0x0012fab4, class nsISupports * aCurrentTarget = 0x03cb1c70, unsigned int aFlags = 6, nsEventStatus * aEventStatus = 0x0012fab8)+0x37f [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\content\events\src\nseventlistenermanager.cpp @ 1184]
15 0012fa40 604f84d4 xul!nsEventTargetChainItem::HandleEvent(class nsEventChainPostVisitor * aVisitor = 0x04eb6ff0, unsigned int aFlags = 0)+0xab [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\content\events\src\nseventdispatcher.cpp @ 211]
16 0012fa6c 604f0735 xul!nsEventTargetChainItem::HandleEventTargetChain(class nsEventChainPostVisitor * aVisitor = 0x00000200, unsigned int aFlags = 0x28246c89, class nsDispatchingCallback * aCallback = 0x8b662474)+0x1e4 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\content\events\src\nseventdispatcher.cpp @ 270]
17 0012fae8 609d4a17 xul!nsEventDispatcher::Dispatch(class nsISupports * aTarget = 0x0282bdd0, class nsPresContext * aPresContext = 0x0282bdd0, class nsEvent * aEvent = 0x0282bdd0, class nsIDOMEvent * aDOMEvent = 0x0284f000, nsEventStatus * aEventStatus = 0x00000e0c, class nsDispatchingCallback * aCallback = 0x00000007)+0x3b5 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\content\events\src\nseventdispatcher.cpp @ 487]
18 0012fb94 60b1d132 xul!nsXULPopupManager::FirePopupShowingEvent(class nsIContent * aPopup = 0x03cb1c70, class nsIContent * aMenu = 0x0012fbb8, class nsPresContext * aPresContext = 0x00636300, nsPopupType aPopupType = ePopupTypeTooltip (2), int aIsContextMenu = 0, int aSelectFirstItem = 0)+0x93 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\layout\xul\base\src\nsxulpopupmanager.cpp @ 1002]
19 0012fbb8 60b1e2e6 xul!nsXULPopupManager::ShowPopupAtScreen(class nsIContent * aPopup = 0x03cb1c70, int aXPos = 593, int aYPos = 642, int aIsContextMenu = 0, class nsIDOMEvent * aTriggerEvent = 0x0280af90)+0x57 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\layout\xul\base\src\nsxulpopupmanager.cpp @ 475]
1a 0012fbf4 60b1e40a xul!nsXULTooltipListener::LaunchTooltip(void)+0x134 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\layout\xul\base\src\nsxultooltiplistener.cpp @ 522]
1b 0012fc34 60b1f026 xul!nsXULTooltipListener::ShowTooltip(void)+0xf6 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\layout\xul\base\src\nsxultooltiplistener.cpp @ 416]
1c 0012fc44 6055aedf xul!nsXULTooltipListener::sTooltipCallback(class nsITimer * aTimer = 0x00652310, void * aListener = 0x0388e7e0)+0x1b [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\layout\xul\base\src\nsxultooltiplistener.cpp @ 749]
1d 0012fc58 6055ae53 xul!nsTimerImpl::Fire(void)+0x7f [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\xpcom\threads\nstimerimpl.cpp @ 400]
1e 0012fc64 604c65c8 xul!nsTimerEvent::Run(void)+0x1f [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\xpcom\threads\nstimerimpl.cpp @ 492]
1f 0012fc88 604b0b8a xul!nsThread::ProcessNextEvent(int mayWait = <Memory access error>, int * result = <Memory access error>)+0x218 [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\xpcom\threads\nsthread.cpp @ 511]
20 0012fca0 60645f7f xul!nsBaseAppShell::Run(void)+0x4a [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\widget\src\xpwidgets\nsbaseappshell.cpp @ 169]
21 0012fcac 60581f75 xul!nsAppStartup::Run(void)+0x1e [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\toolkit\components\startup\src\nsappstartup.cpp @ 182]
22 0012fcb4 0061c0a8 xul!XRE_main(int argc = 0, char ** argv = 0x000005ce, struct nsXREAppData * aAppData = 0x00000009)+0xdba [e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\toolkit\xre\nsapprunner.cpp @ 3174]
WARNING: Frame IP not in any known module. Following frames may be wrong.
23 0012fcb8 00000000 0x61c0a8

0:000> dv
            npp = 0x02c23ad8
             cx = 0x028157c0
          npobj = 0x04eb6ff0
             ar = class JSAutoRequest
0:000> !vprot 0x04eb6ff0
BaseAddress:       04eb6000
AllocationBase:    04e10000
AllocationProtect: 00000001  PAGE_NOACCESS
RegionSize:        0005a000
State:             00001000  MEM_COMMIT
Protect:           00000001  PAGE_NOACCESS
Type:              00020000  MEM_PRIVATE
0:000> ?? npobj->_class
Memory access error at '_class'
0:000> dt npobj
Local var @ 0x12ec40 Type NPObject*
0x04eb6ff0 
   +0x000 _class           : ???? 
   +0x004 referenceCount   : ??
Memory read error 04eb6ff4

Comment 25

[martin]

Attachment: the first BPs from comment #13 while having page heap enabled

Comment 26

[martin]

Attachment: smarter BPs from comment 13

smarter BPs from comment 13, this log shows createobject returning 04f9aff0 and then later (without any release of 04f9aff0 as far as I can see) when 04f9aff0 is used the memory is not accessible ("!vprot 04f9aff0" returns AllocationProtect PAGE_NOACCESS for this address).

Comment 27

[timeless]

:(, I guess the next step is to figure out how to track things that change the protect flags and catch something changing the flag for this memory region.

VirtualProtect  http://msdn.microsoft.com/en-us/library/aa366898(VS.85).aspx
should be the function. it should be possible to calculate in advance the right region to conditionally break for. but if things run fast enough, I'd start with:

bp kernel32!VirtualProtect "kp 3;g"

Comment 28

[aravindm]

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100413 Firefox/3.6.4                                                                     It works for me. Please update if you are able to still reproduce with the latest nightly build  ftp://ftp.mozilla.org/pub/firefox/nightly/latest-trunk/

Comment 29

[Wayne Mery (:wsmwk)]

WFM per comment 28