598679 - Using JS-CTypes to call CallWindowProc crashes XULRunner

Status: RESOLVED DUPLICATE of bug 915613

(Core :: js-ctypes, defect)

Description

[poppertd]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 (CK-zz) Firefox/3.6.8
Build Identifier: XULRunner 2.0b7

I am developing a XUL application that uses js-ctypes to override the main XUL window's WNDPROC to process a few Windows messages (e.g. WM_MOVE).  I am using XBL to write a custom XUL element that calls a JavaScript Module which uses js-ctypes.  In my JSM, I define the following:

    var WindowProcType = ctypes.FunctionType(ctypes.stdcall_abi, ctypes.int, [
			WinDef.HWND,		// hwnd
			ctypes.uint32_t,	// uMsg
			WinDef.WPARAM,		// wParam
			WinDef.LPARAM		// lParam
		]).ptr;
    
    var CallWindowProc = lib.declare("CallWindowProcW", ctypes.winapi_abi, WinDef.LRESULT,
			this.WindowProcType,	// lpPrevWndFunc
			WinDef.HWND,			// hWnd
			ctypes.uint32_t,		// Msg
			WinDef.WPARAM,			// wParam
			WinDef.LPARAM			// lParam
		);

    var SetWindowLongWNDPROC = lib.declare("SetWindowLongW", ctypes.winapi_abi, this.WindowProcType,
			WinDef.HWND,	// hWnd,
			ctypes.int32_t,	// nIndex
			this.WindowProcType	// dwNewLong
		);
    
    var MainWindowProc = function(hwnd, uMsg, wParam, lParam) {
        // Handle some messages
        // ...

        // Call this for messages I don't handle
        return CallWindowProc(hwnd, uMsg, wParam, lParam);
    };

In my code I call:

    g_mainWndProc = WindowProcType(MainWndProc);
    g_lpPrevWndFunc = GetWindowLongWNDPROC(g_hwnd, User32.GWL_WNDPROC);
    SetWindowLongWNDPROC(g_hwnd, User32.GWL_WNDPROC, g_mainWndProc);

My problem is that this causes XULRunner to crash due to an 'Access violation reading location...' and when I attached a debugger to xulrunner.exe, it pointed to Line 792 of jsapi.cpp ( JS_ASSERT(CURRENT_THREAD_IS_ME(cx->thread)); ).

I have a minimal XULRunner application that reproduces this problem.

Reproducible: Always

Steps to Reproduce:
1. Create a XULRunner app that makes the calls to CallWindowProc
2. Mouse over and mouse out of the main XUL window a few times
Actual Results:  
XULRunner crashes

Expected Results:  
XULRunner should have sent the window message on to be processed.

Not all window messages cause a crash.  The application does in fact send some messages on to be processed successfully.

The debugger reports the following error message:

Unhandled exception at 0x0043b492 (xul.dll) in xulrunner.exe: 0xC0000005: Access violation reading location 0x00000008.

Comment 1

[poppertd]

Attachment: Example XUL Application that reproduces the problem

Comment 2

[Benjamin Smedberg]

We need a stacktrace. You can get one by attaching MSVC and using the symbol server: https://developer.mozilla.org/En/Using_the_Mozilla_symbol_server

Comment 3

[poppertd]

Attachment: Stack trace

Comment 4

[[e-mail]]

I still suspect GC here. That stack looks to me like you've got a garbage cx.

  var MyWnd = new function() {
  [...]
        // Global Variables
       
        var g_hwnd;
        var g_lpPrevWndFunc;
        var g_mainWndProc;
I'm not sure (this may be a red herring) but those might be subject to GC once MyWnd() has finished executing. Try sticking it on the global proper, i.e. 'this.g_mainWndProc'?

An easy way to check would be to set the pref 'javascript.options.gczeal' to 2 and restart the browser. It'll be awful slow, but if it's GC related, you'll get a crash immediately rather than at some indeterminate later time. (I'm assuming that the gczeal pref-reading code in nsJSEnvironment is invoked for jsm's, but this may not be true!)

You could also stick a breakpoint in CClosure::Finalize (in CTypes.cpp) and see if it gets invoked, and for what object.

Comment 5

[poppertd]

I tried putting those global variables on global proper as you suggested, but the app still crashes for the same reason.

I also tried setting the pref 'javascript.options.gczeal' to 2, and though it ran slowly, it crashed in the same place (after mousing over/out of the window).

The breakpoints I put in CClosure::Finalize were never reached.

Comment 6

[[e-mail]]

OK. So the cx isn't garbage. I think that rules out GC.

I'm not really sure what this could be. The threading model for WndProcs is single-threaded, and regardless, we know that ctypes closures work fine off-main-thread...