671266 - Crash [@ StartRequest] when calling FuncType callback from native code (WNDPROC)

Status: RESOLVED DUPLICATE of bug 915613

(Core :: js-ctypes, defect)

Description

[Nils Maier [:nmaier]]

Attachment: Crashing extension

This bug was filed from the Socorro interface and is 
report bp-0a70b104-9104-42b4-a767-820732110713 .
============================================================= 
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0

I'm in the process of converting one of my binary components to js-ctypes.
The crash occurs when calling a FuncType.ptr callback function from native code under certain circumstances.

* FuncType.ptr definition:
https://github.com/nmaier/mintrayr/blob/ctypes/modules/platform.jsm#L158
* FuncType.ptr instance:
https://github.com/nmaier/mintrayr/blob/ctypes/modules/platform.jsm#L158
* Corresponding C definition:
https://github.com/nmaier/mintrayr/blob/ctypes/lib_win/tray/tray.h#L7
* Crashing call:
https://github.com/nmaier/mintrayr/blob/ctypes/lib_win/tray/tray.cpp#L148
* Working call (same FuncType.ptr instance):
https://github.com/nmaier/mintrayr/blob/ctypes/lib_win/tray/tray.cpp#L125

Steps to reproduce:
- Install extension from attachment
- Or build yourself: Clone https://github.com/nmaier/mintrayr/tree/ctypes and checkout ctypes branch and build libwin/tray/tray.sln
- Close the window by Alt+F4
- Crash:
|cx->thread| (and hence |t|) is 0 in:
http://mxr.mozilla.org/mozilla-central/source/js/src/jsapi.cpp#835

Expected:
- The call should succeed and the browser window should be put into the tray.

The extension from the attachment contains the dll compiled under a Debug configuration/static crt.

(The extension is a WIP; no need to point out unrelated bugs like incomplete cleanup)

Comment 1

[Bobby Holley (:bholley)]

Taking this bug to get it on my radar. It might take a little while before I get to it though, because I don't currently have a win32 dev setup.

Comment 2

[[e-mail]]

The crash occurs when calling your native WatchedWindow callback function from JS, so it's probably unrelated to the prior native -> JS call.

Where do the WatchedWindows get added? Where do those callback pointers come from? Where is the declaration for the callback type?

Does the native -> JS call consistently work at the one callsite and not work at the other?

Comment 3

[[e-mail]]

(To clarify, it actually occurs immediately after the native call returns, which might indicate an incorrect function declaration.)

Comment 4

[Benjamin Smedberg]

Exactly: your C declaration has WINAPI (which is stdcall), but your ctypes declaration is ctypes.default_abi (which is cdecl). Switch to ctypes.winapi_abi.

Comment 5

[Nils Maier [:nmaier]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> Exactly: your C declaration has WINAPI (which is stdcall), but your ctypes
> declaration is ctypes.default_abi (which is cdecl). Switch to
> ctypes.winapi_abi.

Actually the definition is:
typedef BOOL (*minimize_callback_t)(void *handle, int reason);
No WINAPI there and the default ABI on windows should be cdecl == ctypes.default_abi.

What's crashing in my debugger is a null pointer dereference because |cx->thread == 0|.

(In reply to Dan Witte (:dwitte) from comment #2)
> Where do the WatchedWindows get added? Where do those callback pointers come
> from? Where is the declaration for the callback type?

Sorry, linked the wrong line in comment 0 for the definition. Should be:
https://github.com/nmaier/mintrayr/blob/ctypes/modules/platform.jsm#L23
The callback pointer is always the same ctypes instance, initialized here:
https://github.com/nmaier/mintrayr/blob/ctypes/modules/platform.jsm#L158

> Does the native -> JS call consistently work at the one callsite and not
> work at the other?

Yep, the two callsites (one working, one crashing, same pointer) are stated in comment 0.

Comment 6

[Bobby Holley (:bholley)]

I'm busy enough right now that I'm unlikely to get to this in the immediate future. Updating the assignee to reflect reality. :-(