313443 - Caret move events from keyboard input not reported for XUL textboxes

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[Ian Weiner]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20051021 Firefox/1.6a1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20051021 Firefox/1.6a1

Bug 312941 was prematurely closed. While that fix does generate the proper caret location change events generated by mouse clicks, location changes originated from the keyboard, i.e. inserting text, still don't generate any events in the XUL textboxes. This can be verified with the MSAA Inspector.

In reference to bug 244119 (Tablet PC support), the fix for 312941 now allows the tablet extension to open the floating input panel when the XUL textboxes are clicked, but it automatically closes if the user types any keystroke on the onscreen keyboard and the textbox must be continually re-clicked after each keystroke to use the onscreen keyboard for input.

Reproducible: Always

Steps to Reproduce:
1.Run MSAA Inspect with cursor tracking viewed
2.Select a XUL textbox
3.Enter or change text

Actual Results:  
No cursor location changes are reported

Expected Results:  
Cursor location changes are reported

Comment 1

[Dale Smith]

This bug is still present in thunderbird 1.5.0.2 with geckotip 0.9.3.

Comment 2

[Ian Weiner]

(In reply to comment #1)
> This bug is still present in thunderbird 1.5.0.2 with geckotip 0.9.3.

Dale and any other Geckotip users, this bug is independent of the GeckoTIP extension (although GeckoTIP users might be the only ones affected by it currently). Please don't report GeckoTIP issues in this thread; You can contact me (the author) directly for those. Currently the extension works around this bug with its own checks and you shouldn't be seeing it.

Comment 3

[Aaron Leventhal]

I'm seeing this on trunk when you hit Ctrl+L to get to the location bar. If you shift+tab to the location bar the bug does not occur.

Comment 4

[Aaron Leventhal]

For some reason when focus is currently in the HTML doc, and I hit Ctrl+L, the presShell hasn't yet been set in the nsISelection impl nsTypedSelection.

Comment 5

[Aaron Leventhal]

Attachment: Ensure correct selection attached for each focus event, comments in patch

Comment 6

[Ian Weiner]

Aaron, thanks for looking into this. To clarify, the bug is more severe than just this, though. In my experience, no matter how you initially get the cursor to a XUL box, once you start typing text in it, the expeced cursor change events aren't triggered as text is entered; it's as though the cursor was staying in it's initial position of the textbox no matter what you've typed. 

In the GeckoTIP extension (for Tablet PC support) I've found I can "manually" trigger the cursor events needed by calling 

tbox.setSelectionRange(tbox.selectionStart,tbox.selectionEnd);

as needed when the user types. Hopefully this information helps you in tracking down the bug.

Comment 7

[Aaron Leventhal]

That bigger problem is probably that accessibility is not getting initialized. Try loading Inspect.exe from the MSAA SDK or calling AccessibleObjectFromWindow() on one of the Mozilla windows. That will wake up the accessibility service. Once it wakes up it should start generating events.

Comment 8

[Ian Weiner]

(In reply to comment #7)
> That bigger problem is probably that accessibility is not getting initialized.
> Try loading Inspect.exe from the MSAA SDK or calling
> AccessibleObjectFromWindow() on one of the Mozilla windows. That will wake up
> the accessibility service. Once it wakes up it should start generating events.
> 

Aaron, maybe I'm confused about the accessibility service, I don't know. At least on my machine, if you monitor caret events in inspect, and select a XUL textbox with a mouse click, you DO see caret change events reported from, say, hitting left or right arrow keys, or clicking elsewhere in the box to change the cursor position. Doesn't this alone imply that the accessibility service is loaded at this point and sending events, or am I wrong? The problem is, if you try actually TYPING new text into the box with the keyboard, NO caret change events are noted by inspect. This is in contrast to an HTML textbox on a webpage, which DOES produce the expected caret location change events when you type. I've confirmed this on my Tablet PC and on a non-tablet, both running Firefox 1.5.0.4. Can you reproduce this behavior or clarify my understanding of the situation?

Comment 9

[Aaron Leventhal]

Okay, I understand better now -- my fault I'm getting confused by dealing with too many bugs at once.

Comment 10

[Aaron Leventhal]

I see the problem with typing text, the caret is hidden temporarily when text is typed, presumably to avoid layout residue. We suppress location changed events for the caret when the caret is hidden.

The easy fix would be to just fire the event even if the caret is hidden. Ian, would that work for you? The caret location would also be reported as moved when the user tabs to a new link, because there is a hidden caret that moves around when you tab.

Comment 11

[Aaron Leventhal]

Attachment: Also shows MSAA caret move events for hidden caret

Comment 12

[Ian Weiner]

    (In reply to comment #10)
    > The easy fix would be to just fire the event even if the caret is hidden. Ian,
    > would that work for you? The caret location would also be reported as moved
    > when the user tabs to a new link, because there is a hidden caret that moves
    > around when you tab.

    Sounds ok to me as long as nothing gets inadvertantly broken or weird as a result. In fact it sounds like it may even fix a few other oddities for Tablet users here and there, like events not firing sometimes when textboxes get selected programmatically. Unfortunately, my Tablet PC is currently away for repairs (and the company resposible for it seems to be less than competant) so I can't test any fixes for a little while.

    Another question, related to bug 244119: Could you please elaborate on the caret(s) implemented in Mozilla/Firefox? In the GeckoTIP extension I create my own invisible Win32 caret and use your accessibility events to move it so it tracks with Firefox's regular caret. This invisible caret is what the TIP automatically tracks and it's what enables the Tablet support. ANY Win32 caret made with CreateCaret, properly updated as necessary SetCaretPos, and destroyed with DestroyCaret, is enough for the TIP to interface with an application without any extra custom code for Tablet PCs. This is why pretty much all native windows applications work with the TIP without any Tablet-specific code whatsoever. Is there a Win32 caret implemented in the Mozilla codebase already someplace, perhaps used in conjunction with Accessibility? If so, it may be that Mozilla already more or less supports the TIP natively but there's just a bug here and there preventing it from fully working properly. The reason I ask is that even without any extensions, Firefox seems to work sporadically with the TIP, which it shouldn't at all unless there's some sort of windows native carets in there *somewhere*. Maybe full Tablet support can be added with just a bugfix or minor tweak? Or am I totally off-base here?

Comment 13

[Aaron Leventhal]

Firefox doesn't have any native carets. It's all screen draws.

Comment 14

[Ian Weiner]

(In reply to comment #13)
> Firefox doesn't have any native carets. It's all screen draws.

Ok, maybe the partial functionality has something to do with the TIP tracking something else, then...

Even so, I just wanted to point out that Bug 244119 can be completely resolved if you can add a few Win32 caret calls alongside your code that generates caret change events for Accessibility. All you need is CreateCaret, SetCaretPos, and DestroyCaret, mirroring what the onscreen caret does. I'm not familiar with the codebase-- is this feasible with only small changes? If it is, I can try to modify my GeckoTIP code and submit the patch myself, if you can direct me to the right places.

Comment 15

[Aaron Leventhal]

What is better about using the system caret location over MSAA caret locationchange events?

Comment 16

[Ian Weiner]

(In reply to comment #15)
> What is better about using the system caret location over MSAA caret
> locationchange events?

The MSAA caret locationchange events by themselves don't do anything that triggers the Tablet Input Panel (TIP), so the GeckoTIP extension is required. The main part of the extension essentially just creates an invisible Win32 caret and monitors the MSAA events Firefox sends out, updating its caret based on the location change events, etc. The TIP automatically monitors and uses this info to mediate input with the application. If you could just create the Win32 caret in Firefox itself, the Tablet Input Panel would work with Firefox and the other Mozilla applications without any extensions required. By the way, MS's new UMPC/Origami initiative also runs Tablet XP, so it would work with those, too.

So if it's not hard to do, it'd be better to have the built-in Tablet PC support in all Mozilla apps, rather than having to install an extension. If you google for "firefox tablet pc" or read some Tablet PC message boards, you'll see that many users just open Firefox, see the TIP doesn't work, and assume it can't be helped (and use IE instead). Also, extensions can break with new versions of Firefox (aside from just the MaxVersion checking); The switch from Firefox 1.0 to 1.5 broke GeckoTIP's predecessor, TPCTIP, and the C code needed to be relinked because of changes to Mozilla's XPCOM.DLL file. The original author was MIA, and that's how I got involved in this stuff in the first place.

Comment 17

[Aaron Leventhal]

It sounds like this discussion belongs in bug 244119. Anyway, I think it might be okay to do what you say, but better not in the mozilla/accessible code. It should be somewhere in nsCaret.cpp or widget/src/windows or someplace that listens to caret move events. Mozilla/accessible just happens to be one listener there, and it's often not loaded. Plus it requires extra memory overheard to use the accessibility module.

Comment 18

[Ginn Chen]

Comment on attachment 228999 [details] [diff] [review]
Also shows MSAA caret move events for hidden caret

align the comment line

+  t2p = presContext->TwipsToPixels();
+    // Convert to pixels using that scale
+  caretRect.x      = NSTwipsToIntPixels(caretRect.x, t2p);

Comment 19

[Aaron Leventhal]

Ian, should we try to get this into Firefox 2? The hidden caret move events won't affect screen readers because they don't use the MSAA caret, so no risk there. Is there any chance this could make things worse for tablet PC?

Comment 20

[Ian Weiner]

(In reply to comment #19)
> Ian, should we try to get this into Firefox 2? The hidden caret move events
> won't affect screen readers because they don't use the MSAA caret, so no risk
> there. Is there any chance this could make things worse for tablet PC?

How much time do we have to work on this? I'd say there's always a chance it could somehow make things worse, but it wouldn't take long to test thoroughly enough to find any serious issues. The problem on my end is my Tablet is gone for repairs.  I think I could find some GeckoTIP users willing to test for us, but it's not ideal.  

Comment 21

[Aaron Leventhal]

Probably a couple of weeks, but it makes me nervous to wait. The users can download one of the unstable trunk builds here (starting tomorrow afternoon it will have this fix)
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

Comment 22

[Ian Weiner]

Aaron, based on my tester's report that functionality improved, I'll formally recommend that this patch be included in the Firefox 2 branch. (and even if something weird shows up, I can probably work around it again in the extension code.)

Comment 23

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 228999 [details] [diff] [review]
Also shows MSAA caret move events for hidden caret

a=dbaron on behalf of drivers.  Please check in to the MOZILLA_1_8_BRANCH and mark fixed1.8.1 when you have.

Comment 24

[Robert "Unlogic" Olofsson]

This bug is still present in the Firefox 2.0 search field with Gecktip 0.9.6

Comment 25

[Gustav Wall]

This bug is still present in the Firefox 3.0 search field and location bar with GeckoTip 1.0.1

Comment 26

[Ian Weiner]

Please note that this bugfix will NOT help Tablet PC users running Windows Vista, which seems to not use MSAA events to track input fields as XP does.

The gesso extension (http://gesso.mozdev.org) by Jim Chen adds Text Services Framework support to Mozilla, a more robust way to get the Tablet PC input panel communicating with Mozilla input fields that works for XP and Vista. Please see bug 88831 for more information as well. Gesso is in beta but works relatively bug-free.