484050 - Windows 7 raw touch events not available to plugins

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect, P1)

Description

[Jakub Plichta]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 GTB5 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 GTB5 (.NET CLR 3.5.30729)

I am writing a plugin which provides direct multi-touch manipulation on Windows 7. In the setwindow API function I call RegisterTouchWindow from Win 7 SDK to enable touch events. In my winproc I look for WM_TOUCHDOWN, WM_TOUCHUP and WM_TOUCHMOVE. They never show up.

Reproducible: Always

Steps to Reproduce:
1. Create a simple plugin
2. RegisterTouchWindow on the HWND from NPWindow passed to setwindow.
3. Register winproc, add code to process WM_TOUCHDOWN events



Actual Results:  
Gestures (pinch) stop working on the plugin window but the touch events are not forwarded to the winproc.

Expected Results:  
Gestures stop working and the workproc starts receiving touch events.

I suspect that touch events have to be added to the message map for the plugin window.

Comment 1

[Jim Mathies [:jimm]]

Is this an issue on windowed or windowless plugins? Sounds like windowed from the comments but I want to be sure. I'm building a debug test plugin on a tablet now and will see if I can reproduce for a _windowed_ plugin.

Comment 2

[Jim Mathies [:jimm]]

Hmm, just poking around we have a few windows message hooks in widget but overall I don't see anything that would cause this for windowed controls. Maybe this is with windowless controls? (I'll take a look at both.)

Comment 3

[Jakub Plichta]

we support both modes. This issue was encountered in windowed mode and I have not tested windowless. I can give it a try later this afternoon.

Comment 4

[Jim Mathies [:jimm]]

(In reply to comment #3)
> we support both modes. This issue was encountered in windowed mode and I have
> not tested windowless. I can give it a try later this afternoon.

In windowed mode you're setting a subclass on the plugin child window, and these events simply don't show up in your wndproc?

Comment 5

[Jim Mathies [:jimm]]

Looks like in windowless the events that come in through handleevent are heavily filtered. I think that's by design but I'll ask around, we might be able to add the touch events to the smallish list of windows events we send. Seems though that most developers are going to use windowed so I'm not sure it's needed.

Windowed is what I'm looking at now. Overall not seeing much of anything by just surveying code. We do have some message filters down in our windows widget code but I'm not seeing anything yet that would get in the way of this. Going to do some debugging once my test build finishes up on the tablet.

Comment 6

[jossmit]

FYI - we are reproing this bug in windowed mode only...you can't call RegisterTouchWindow on a windowless window, so we aren't concerned about the windowless scenario at the moment.

We subclass the plugin child window's wndproc, but WM_TOUCH messages are not making it our wndproc.

Comment 7

[Jim Mathies [:jimm]]

Attachment: nptest_windows.cpp

Attached is a drop in replacement for 

/modules/plugin/testplugin/nptest_windows.cpp

In it I've added the necessary code for touch support and registered the test plugin hwnd for touch input. I set a break point in wind proc in the case statement for WM_TOUCHDOWN, WM_TOUCHMOVE, & WM_TOUCHUP. When I hold my finger down on the plugin window, I get three breaks, one for each of the touch events, so so far I can't reproduce. These events seem to be working fine. Can you all look this over and see if you can spot any obvious differences in how you're handling things on your end?

Comment 8

[Jim Mathies [:jimm]]

I should add - I'm testing on the latest public win7 beta (build 7000), on an HP TouchSmart tablet laptop using n-Trig drivers released about a month ago (driver version 1.26).

Comment 9

[Jim Mathies [:jimm]]

Another data point, I just checked the same test plugin in Shiretoko (3.5 release) and found the events working as well. Sol I've checked 1.9.2 and 1.9.1 and both are working. Possibly this is problem in 3.0 that has already been addresses. 3.0 is hard for me to test due to changes in our plugin test framework. If you're having problems with these events in 3.0, please download one of the latest 3.5 builds and test with that release. If you're having trouble there, please post back.

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009-04-17-04-mozilla-1.9.1/

Comment 10

[Jim Mathies [:jimm]]

please re-open if you're still experiencing problems on Fx 3.5.

Comment 11

[Jakub Plichta]

Jim, thank you for checking this out. I am reproducing the bug in Shirotoko 2009-04-21-04-mozilla-1.9.1.

The behavior is really strange and I do not understand how it could be caused by the browser.

Here is the sequence of events:
1. ::SetWindowLongPtr( hWnd, GWLP_WNDPROC, platformPlayerProc );
2. RegisterTouchWindow( hWnd, 0 );
3. conditional break point in platformPlayerProc for message==0x0240 never triggers.

Weird thing is that I can see in Spy++ that the 0x240 messages are going to my window. So I know that RegisterTouchWindow worked.

Also WM_GESTURE messages are forwarded to the window proc correctly.

I am not aware of anything that could be preventing the touch messages from reaching the window proc.

I think the next step would be to get the project files for your sample plugin with instructions how to get it loaded into 3.5. Either it will work and the issue is somewhere on our end or it will fail and the bug is in windows or browser. 

BTW I am running Win 7 build 7068 which is fairly recent. Similar code in IE 8 works fine.

Comment 12

[Jim Mathies [:jimm]]

(In reply to comment #11)
> Jim, thank you for checking this out. I am reproducing the bug in Shirotoko
> 2009-04-21-04-mozilla-1.9.1.
> 
> The behavior is really strange and I do not understand how it could be caused
> by the browser.
> 
> Here is the sequence of events:
> 1. ::SetWindowLongPtr( hWnd, GWLP_WNDPROC, platformPlayerProc );
> 2. RegisterTouchWindow( hWnd, 0 );
> 3. conditional break point in platformPlayerProc for message==0x0240 never
> triggers.
> 
> Weird thing is that I can see in Spy++ that the 0x240 messages are going to my
> window. So I know that RegisterTouchWindow worked.
> 
> Also WM_GESTURE messages are forwarded to the window proc correctly.
> 
> I am not aware of anything that could be preventing the touch messages from
> reaching the window proc.
> 
> I think the next step would be to get the project files for your sample plugin
> with instructions how to get it loaded into 3.5. Either it will work and the
> issue is somewhere on our end or it will fail and the bug is in windows or
> browser. 
> 
> BTW I am running Win 7 build 7068 which is fairly recent. Similar code in IE 8
> works fine.

Rather than have you go through the hassle of building 1.9.1 and our test framework / plugin, I'll work up a stand alone test plugin project to work with.

One thing I noticed is that I'm registering the touch window before I set the subclass. Doubt that has any effect but who knows.

Comment 13

[Jakub Plichta]

I reversed the order but no difference. Let me know if you can think of anything else I could try. For now I am waiting for a sample plugin.

Comment 14

[Jim Mathies [:jimm]]

I've been able to reproduce this in a stand alone plugin I built from scratch. So I now have two plugins, both of which are basically the same except that they are built in slightly different ways. (the first, our test framework plugin, is build using our make environment, and the second using visual studio.) Unfortunately I haven't found the reason the second isn't working.

This seems to have something to do with calling RegisterTouchWindow in the plugin. In the case of the plugin that doesn't work, we receive the events in our main message pump and fire them off via translatemessage and dispatchmessage, at which point they disappear into the ether.  In the plugin that works, immediately after dispatchmessage is called the event shows up in the window subclass of the plugin.

Both of these plugins are built using the current sdk headers and are running in the same version of firefox. One is a bit more advanced as it's used in our testing, the other is a bare bones plugin I built from scratch using useful code from the first. Bosh seem to follow the same set of steps during creation initialization.

I wish I had the code to dispatchmessage so I could figure out why these events are being filtered on the window that calls RegisterTouchWindow, but I don't. I'll keep poking around at it. If anyone would like my samples just ask.

Generally though it doesn't appear to be anything in firefox at this point.

Comment 15

[Jim Mathies [:jimm]]

Some updates on the above post. Fx does set a sublass on the plugin window, so technically the message is lost on the first call to CallWndProc in the Fx plugin subclass.

- The behavior is interesting, there are two subclasses and then the main window procedure. With plugins that work the call chain looks like this:

DispatchMessage
PluginNative subclass -> CallWndProc
Plugin subclass -> CallWndProc
Base nsWindow wndproc

On those that don't, messages get filtered on the plugin subclass:

DispatchMessage
PluginNative subclass -> CallWndProc
*never called* Plugin subclass -> CallWndProc
*never called* Base nsWindow wndproc

So the filtering is happening on the call from our subclass into the plugin subclass.

Removing the plugin subclass allows these events to propagate through the call chain if the window is registered.

It's also interesting to note that the plugin reports the window is registered via IsTouchWindow, even though these messages get filtered.

In general, it still seems as though these messages are being filtered due to some unknown problem windows is having with the subclass procedure in the plugin.

Joe, any way you guys could help us out with some debugging?

Comment 16

[jossmit]

(I'm assuming I'm the Joe that you are referring to.)

We aren't legally allowed to look at any source...but we can try looking at the repro in debugger with win source - I'll keep you posted.

Comment 17

[Jim Mathies [:jimm]]

Attachment: test stand alone plugin

Here's the test stand alone plugin I'm working with. I'm out in Mtn View this week so I don't have access to my tablet to test. I'll come back to this next week.

Comment 18

[jossmit]

Thanks - I've passed the repro onto the touch developers - they are looking into it.

Comment 19

[jossmit]

Hey Jim - is there any UNICODE/ANSI differences between your two projects (the one that works and the one that doesn't?)

Comment 20

[Jim Mathies [:jimm]]

Attachment: updated test plugin

Joe - turns out MB vs. UNICODE was the issue. The multibyte build works, the unicode one does not. Here's an updated stand alone test plugin that builds cleanly as both. Any ideas as to why unicode plugins get the filtering?

Comment 21

[jossmit]

Yes - we've also confirmed this issue and have a fix. It is currently being considered for Windows 7 RTM. Thanks a bunch for the help.

Comment 22

[Jim Mathies [:jimm]]

(In reply to comment #21)
> Yes - we've also confirmed this issue and have a fix. It is currently being
> considered for Windows 7 RTM. Thanks a bunch for the help.

Joe, if for some reason it doesn't make it in, we at least know we have a work around. Please post back if your fix makes it into the final release so we can close this bug out. We appreciate all the help from you and the folks in Redmond.

Jakup, many thanks to you and Adobe for posting this, your help, and staying on top of the problem.

Comment 23

[Jakub Plichta]

Big thank you for everyone at Mozilla and Microsoft who help track and resolve this issue. The Flash player has to be consistent across browser and it would be a shame if we could not support MT because of this issue.

Comment 24

[Jakub Plichta]

do we have an update regarding the status of this bug at MSFT? Will the fix make it into Win 7? Thanks!

Comment 25

[jossmit]

The fix just made into the main windows 7 builds this week, so it is 90% likely to ship in windows assuming there are no major regressions found last minute.

Thanks again everyone for your help.

Comment 26

[Jakub Plichta]

I can confirm that this bug has been fixed in Windows7 build 7137.

Big thanks to all involved.

Comment 27

[Jim Mathies [:jimm]]

(In reply to comment #26)
> I can confirm that this bug has been fixed in Windows7 build 7137.
> 
> Big thanks to all involved.

Thanks guys!