Caret move events from keyboard input not reported for XUL textboxes

RESOLVED FIXED

Status

()

Core
Disability Access APIs
RESOLVED FIXED
12 years ago
9 years ago

People

(Reporter: Ian Weiner, Assigned: Aaron Leventhal)

Tracking

({access, fixed1.8.1})

Trunk
x86
All
access, fixed1.8.1
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: Does not affect screen readers. Fixes tablet PC functionality. Reports MSAA caret movement even for hidden carets.)

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

12 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20051021 Firefox/1.6a1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20051021 Firefox/1.6a1

Bug 312941 was prematurely closed. While that fix does generate the proper caret location change events generated by mouse clicks, location changes originated from the keyboard, i.e. inserting text, still don't generate any events in the XUL textboxes. This can be verified with the MSAA Inspector.

In reference to bug 244119 (Tablet PC support), the fix for 312941 now allows the tablet extension to open the floating input panel when the XUL textboxes are clicked, but it automatically closes if the user types any keystroke on the onscreen keyboard and the textbox must be continually re-clicked after each keystroke to use the onscreen keyboard for input.

Reproducible: Always

Steps to Reproduce:
1.Run MSAA Inspect with cursor tracking viewed
2.Select a XUL textbox
3.Enter or change text

Actual Results:  
No cursor location changes are reported

Expected Results:  
Cursor location changes are reported

Comment 1

11 years ago
This bug is still present in thunderbird 1.5.0.2 with geckotip 0.9.3.
(Reporter)

Comment 2

11 years ago
(In reply to comment #1)
> This bug is still present in thunderbird 1.5.0.2 with geckotip 0.9.3.

Dale and any other Geckotip users, this bug is independent of the GeckoTIP extension (although GeckoTIP users might be the only ones affected by it currently). Please don't report GeckoTIP issues in this thread; You can contact me (the author) directly for those. Currently the extension works around this bug with its own checks and you shouldn't be seeing it.
(Assignee)

Updated

11 years ago
Blocks: 333492
(Assignee)

Comment 3

11 years ago
I'm seeing this on trunk when you hit Ctrl+L to get to the location bar. If you shift+tab to the location bar the bug does not occur.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(Assignee)

Comment 4

11 years ago
For some reason when focus is currently in the HTML doc, and I hit Ctrl+L, the presShell hasn't yet been set in the nsISelection impl nsTypedSelection.
(Assignee)

Updated

11 years ago
Blocks: 244119
(Assignee)

Comment 5

11 years ago
Created attachment 228979 [details] [diff] [review]
Ensure correct selection attached for each focus event, comments in patch
Attachment #228979 - Flags: superreview?(neil)
Attachment #228979 - Flags: review?(ginn.chen)
(Reporter)

Comment 6

11 years ago
Aaron, thanks for looking into this. To clarify, the bug is more severe than just this, though. In my experience, no matter how you initially get the cursor to a XUL box, once you start typing text in it, the expeced cursor change events aren't triggered as text is entered; it's as though the cursor was staying in it's initial position of the textbox no matter what you've typed. 

In the GeckoTIP extension (for Tablet PC support) I've found I can "manually" trigger the cursor events needed by calling 

tbox.setSelectionRange(tbox.selectionStart,tbox.selectionEnd);

as needed when the user types. Hopefully this information helps you in tracking down the bug.
(Assignee)

Comment 7

11 years ago
That bigger problem is probably that accessibility is not getting initialized. Try loading Inspect.exe from the MSAA SDK or calling AccessibleObjectFromWindow() on one of the Mozilla windows. That will wake up the accessibility service. Once it wakes up it should start generating events.
(Reporter)

Comment 8

11 years ago
(In reply to comment #7)
> That bigger problem is probably that accessibility is not getting initialized.
> Try loading Inspect.exe from the MSAA SDK or calling
> AccessibleObjectFromWindow() on one of the Mozilla windows. That will wake up
> the accessibility service. Once it wakes up it should start generating events.
> 

Aaron, maybe I'm confused about the accessibility service, I don't know. At least on my machine, if you monitor caret events in inspect, and select a XUL textbox with a mouse click, you DO see caret change events reported from, say, hitting left or right arrow keys, or clicking elsewhere in the box to change the cursor position. Doesn't this alone imply that the accessibility service is loaded at this point and sending events, or am I wrong? The problem is, if you try actually TYPING new text into the box with the keyboard, NO caret change events are noted by inspect. This is in contrast to an HTML textbox on a webpage, which DOES produce the expected caret location change events when you type. I've confirmed this on my Tablet PC and on a non-tablet, both running Firefox 1.5.0.4. Can you reproduce this behavior or clarify my understanding of the situation?
(Assignee)

Comment 9

11 years ago
Okay, I understand better now -- my fault I'm getting confused by dealing with too many bugs at once.
(Assignee)

Comment 10

11 years ago
I see the problem with typing text, the caret is hidden temporarily when text is typed, presumably to avoid layout residue. We suppress location changed events for the caret when the caret is hidden.

The easy fix would be to just fire the event even if the caret is hidden. Ian, would that work for you? The caret location would also be reported as moved when the user tabs to a new link, because there is a hidden caret that moves around when you tab.
(Assignee)

Comment 11

11 years ago
Created attachment 228999 [details] [diff] [review]
Also shows MSAA caret move events for hidden caret
Attachment #228979 - Attachment is obsolete: true
Attachment #228999 - Flags: review?(ginn.chen)
Attachment #228979 - Flags: superreview?(neil)
Attachment #228979 - Flags: review?(ginn.chen)
(Reporter)

Comment 12

11 years ago
    (In reply to comment #10)
    > The easy fix would be to just fire the event even if the caret is hidden. Ian,
    > would that work for you? The caret location would also be reported as moved
    > when the user tabs to a new link, because there is a hidden caret that moves
    > around when you tab.

    Sounds ok to me as long as nothing gets inadvertantly broken or weird as a result. In fact it sounds like it may even fix a few other oddities for Tablet users here and there, like events not firing sometimes when textboxes get selected programmatically. Unfortunately, my Tablet PC is currently away for repairs (and the company resposible for it seems to be less than competant) so I can't test any fixes for a little while.

    Another question, related to bug 244119: Could you please elaborate on the caret(s) implemented in Mozilla/Firefox? In the GeckoTIP extension I create my own invisible Win32 caret and use your accessibility events to move it so it tracks with Firefox's regular caret. This invisible caret is what the TIP automatically tracks and it's what enables the Tablet support. ANY Win32 caret made with CreateCaret, properly updated as necessary SetCaretPos, and destroyed with DestroyCaret, is enough for the TIP to interface with an application without any extra custom code for Tablet PCs. This is why pretty much all native windows applications work with the TIP without any Tablet-specific code whatsoever. Is there a Win32 caret implemented in the Mozilla codebase already someplace, perhaps used in conjunction with Accessibility? If so, it may be that Mozilla already more or less supports the TIP natively but there's just a bug here and there preventing it from fully working properly. The reason I ask is that even without any extensions, Firefox seems to work sporadically with the TIP, which it shouldn't at all unless there's some sort of windows native carets in there *somewhere*. Maybe full Tablet support can be added with just a bugfix or minor tweak? Or am I totally off-base here?
(Assignee)

Comment 13

11 years ago
Firefox doesn't have any native carets. It's all screen draws.
(Reporter)

Comment 14

11 years ago
(In reply to comment #13)
> Firefox doesn't have any native carets. It's all screen draws.

Ok, maybe the partial functionality has something to do with the TIP tracking something else, then...

Even so, I just wanted to point out that Bug 244119 can be completely resolved if you can add a few Win32 caret calls alongside your code that generates caret change events for Accessibility. All you need is CreateCaret, SetCaretPos, and DestroyCaret, mirroring what the onscreen caret does. I'm not familiar with the codebase-- is this feasible with only small changes? If it is, I can try to modify my GeckoTIP code and submit the patch myself, if you can direct me to the right places.

(Assignee)

Comment 15

11 years ago
What is better about using the system caret location over MSAA caret locationchange events?
(Reporter)

Comment 16

11 years ago
(In reply to comment #15)
> What is better about using the system caret location over MSAA caret
> locationchange events?

The MSAA caret locationchange events by themselves don't do anything that triggers the Tablet Input Panel (TIP), so the GeckoTIP extension is required. The main part of the extension essentially just creates an invisible Win32 caret and monitors the MSAA events Firefox sends out, updating its caret based on the location change events, etc. The TIP automatically monitors and uses this info to mediate input with the application. If you could just create the Win32 caret in Firefox itself, the Tablet Input Panel would work with Firefox and the other Mozilla applications without any extensions required. By the way, MS's new UMPC/Origami initiative also runs Tablet XP, so it would work with those, too.

So if it's not hard to do, it'd be better to have the built-in Tablet PC support in all Mozilla apps, rather than having to install an extension. If you google for "firefox tablet pc" or read some Tablet PC message boards, you'll see that many users just open Firefox, see the TIP doesn't work, and assume it can't be helped (and use IE instead). Also, extensions can break with new versions of Firefox (aside from just the MaxVersion checking); The switch from Firefox 1.0 to 1.5 broke GeckoTIP's predecessor, TPCTIP, and the C code needed to be relinked because of changes to Mozilla's XPCOM.DLL file. The original author was MIA, and that's how I got involved in this stuff in the first place.

(Assignee)

Comment 17

11 years ago
It sounds like this discussion belongs in bug 244119. Anyway, I think it might be okay to do what you say, but better not in the mozilla/accessible code. It should be somewhere in nsCaret.cpp or widget/src/windows or someplace that listens to caret move events. Mozilla/accessible just happens to be one listener there, and it's often not loaded. Plus it requires extra memory overheard to use the accessibility module.

Comment 18

11 years ago
Comment on attachment 228999 [details] [diff] [review]
Also shows MSAA caret move events for hidden caret

align the comment line

+  t2p = presContext->TwipsToPixels();
+    // Convert to pixels using that scale
+  caretRect.x      = NSTwipsToIntPixels(caretRect.x, t2p);
Attachment #228999 - Flags: review?(ginn.chen) → review+
(Assignee)

Updated

11 years ago
Keywords: access
(Assignee)

Updated

11 years ago
Attachment #228999 - Flags: superreview?(neil)

Updated

11 years ago
Attachment #228999 - Flags: superreview?(neil) → superreview+
(Assignee)

Comment 19

11 years ago
Ian, should we try to get this into Firefox 2? The hidden caret move events won't affect screen readers because they don't use the MSAA caret, so no risk there. Is there any chance this could make things worse for tablet PC?
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
(Reporter)

Comment 20

11 years ago
(In reply to comment #19)
> Ian, should we try to get this into Firefox 2? The hidden caret move events
> won't affect screen readers because they don't use the MSAA caret, so no risk
> there. Is there any chance this could make things worse for tablet PC?

How much time do we have to work on this? I'd say there's always a chance it could somehow make things worse, but it wouldn't take long to test thoroughly enough to find any serious issues. The problem on my end is my Tablet is gone for repairs.  I think I could find some GeckoTIP users willing to test for us, but it's not ideal.  
(Assignee)

Comment 21

11 years ago
Probably a couple of weeks, but it makes me nervous to wait. The users can download one of the unstable trunk builds here (starting tomorrow afternoon it will have this fix)
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
(Reporter)

Comment 22

11 years ago
Aaron, based on my tester's report that functionality improved, I'll formally recommend that this patch be included in the Firefox 2 branch. (and even if something weird shows up, I can probably work around it again in the extension code.)
(Assignee)

Updated

11 years ago
Attachment #228999 - Flags: approval1.8.1?
(Assignee)

Updated

11 years ago
Whiteboard: Does not affect screen readers. Fixes tablet PC functionality. Reports MSAA caret movement even for hidden carets.
Comment on attachment 228999 [details] [diff] [review]
Also shows MSAA caret move events for hidden caret

a=dbaron on behalf of drivers.  Please check in to the MOZILLA_1_8_BRANCH and mark fixed1.8.1 when you have.
Attachment #228999 - Flags: approval1.8.1? → approval1.8.1+
(Assignee)

Updated

11 years ago
Keywords: fixed1.8.1
This bug is still present in the Firefox 2.0 search field with Gecktip 0.9.6

Comment 25

9 years ago
This bug is still present in the Firefox 3.0 search field and location bar with GeckoTip 1.0.1
(Reporter)

Comment 26

9 years ago
Please note that this bugfix will NOT help Tablet PC users running Windows Vista, which seems to not use MSAA events to track input fields as XP does.

The gesso extension (http://gesso.mozdev.org) by Jim Chen adds Text Services Framework support to Mozilla, a more robust way to get the Tablet PC input panel communicating with Mozilla input fields that works for XP and Vista. Please see bug 88831 for more information as well. Gesso is in beta but works relatively bug-free.
You need to log in before you can comment on or make changes to this bug.