Closed Bug 244119 Opened 21 years ago Closed 16 years ago

"In Place Tablet Input Panel" icon doesn't appear for text input

Categories

(Firefox :: Shell Integration, enhancement, P4)

Product:
Firefox
Component:
Shell Integration
Platform:
x86
Windows XP
Type:
enhancement

Tracking

()

Status:
RESOLVED DUPLICATE of bug 286072

People

(Reporter: ryanaip, Unassigned)

References

Details

(Keywords: helpwanted)

Attachments

(2 obsolete files)

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8 Service pack 2 for Windows XP (currently in beta) provides an update to the pen-input functionality of Windows XP Tablet PC edition. In supported applications, an "in place tablet input panel" icon appears/hovers next to any text input control when the tablet pen hovers over it. Clicking on the icon brings up a floating input panel that allows the user to input text directly into the text box. This is likely because Firefox doesn't include support for the Microsoft Active Accessibility API, which is used to track the input caret in text boxes. For more information, see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dntablet/html/carettrack.asp Reproducible: Always Steps to Reproduce: 1. Install Windows XP SP2 beta on a tablet pc (http://www.microsoft.com/technet/prodtechnol/winxppro/sp2preview.mspx) 2. Hover the pen over a text input area (e.g., the address bar) Actual Results: No text input panel icon appeared. Expected Results: The input panel icon should have appeared next to the insertion point caret. (See, for example, the Internet Explorer address bar after installing the service pack.)
Can you test with a recent nightly (mozilla or firefox) ? (1.6 is old) This is probably a issue with Mozilla, not just Firefox
Tested again with the latest nightly of Mozilla (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040521) The same issue occurrs. When the program first launches, the icon shows up once or twice (clicking on either the address bar or the search box on the default home page - http://www.mozilla.org/start/). A couple of times, I've seen it appear very briefly and then disappear. So, it could be that mozilla is rendering on top of the icon. Otherwise, it's possible that the text caret position has an initial default that's never being updated.
This bug happens for me in both firefox and thunderbird
Firefox 0.2.9 displays same behavior under Windows XP Tablet PC Edition Service Pack 2 Version 2149
(In reply to comment #4) > Firefox 0.2.9 displays same behavior under Windows XP Tablet PC Edition Service > Pack 2 Version 2149 Same problem on my Tablet. I'm using Firefox 0.9.2 & Thunderbird 0.7.2
"Firefox doesn't include support for the Microsoft Active Accessibility API, which is used to track the input caret in text boxes." Aaron, is this something you can help us with?
Assignee: bugs → aaronleventhal
(In reply to comment #6) > "Firefox doesn't include support for the Microsoft Active Accessibility API, > which is used to track the input caret in text boxes." Yes we do have MSAA support in Firefox! Here are the docs: http://www.mozilla.org/projects/ui/accessibility/vendors-win.html Let me know if something's not to spec, we have Gecko working with Window-Eyes which uses MSAA for almost everything.
To verify that we support MSAA, load firefox and accexplore.exe or inspect.exe from the MSAA SDK tools available on http://www.microsoft.com/downloads/details.aspx?FamilyId=4179742F-1F3D-4115-A8BA-2F7A6022B533&displaylang=en
I just thought I'd add that this issue remains in service pack 2 final running firefox 0.9.3
I've just noticed that the TIP appears correctly on the first page loaded in the browser, but stops working as soon as you go to another page. Pressng f7 to enable/disable caret browsing doesn't help.
(In reply to comment #8) > To verify that we support MSAA, load firefox and accexplore.exe or inspect.exe > from the MSAA SDK tools available on > http://www.microsoft.com/downloads/details.aspx?FamilyId=4179742F-1F3D-4115-A8BA-2F7A6022B533&displaylang=en As other have noted, it works sporadically in Mozilla with the released SP2. Maybe your MSAA code is correct and something else is stomping an it? For the record, here's what Microsoft told me about the problem. "The TIP uses caret tracking events to determine which text input control has focus, and can invoke the Input Panel. The floating TIP icon will not appear when a caret is not present or if caret events are not being accurately generated. (for more information about what is required to enable the floating TIP icon within an application, see this document: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dntablet/html/carettrack.asp )" I'm sure three would be a lot of appreciative tablet users, if someone could take a look at this problem.
This bug appears in Mozilla 1.7.x and Thunderbird as well, So perhaps it should be expanded from just "firefox"? The inability to use an un-docked TIP in Mozilla is a much bigger inconvenience than you might expect. Performing input on only the default top and bottom- docked TIP can give you serious hand Cramps. I have had to revert to using IE and Outlook when using my work Tablet (Gasp!). Perhaps the priority on this bug could be bumped up?
I completely agree. I also have reverted to using IE on my tablet (no worries, I still use Firefox on my desktop). It just makes life much easier for me.
Just tried the pre release 1.0 firefly version . Sorry it's the same. I am going to uninstall it and use IE with my tablet PC. It's a shame that you guys don't support tablet PC users any better.
Just tried the pre release 1.0 firefly version . Sorry it's the same. I am going to uninstall it and use IE with my tablet PC. It's a shame that you guys don't support tablet PC users any better.
sorry, I did not mean Firefly (although I really liked the series, shame it was cancelled), I was referring to the new Firefox preview release candidate 1.0. Sorry for posting it twice, was my first post here.
Someone said it works on the first page. I don't really know what could be wrong. We fire a caret move event. When I run accevent.exe from the MSAA SDK it reports them, as well as the location of the caret. It's working beyond the first page. Can Microsoft point us to a documentation page that describes exactly what calls they use with the MSAA caret? Maybe that will help.
Priority: -- → P4
Will this help? http://msdn.microsoft.com/library/en-us/dntablet/html/carettrack.asp To support Input Panel, you must implement the following five functions, which represent the core caret tracking functionality in MSAA: CreateCaret creates a new shape for the system caret and assigns ownership of the caret to the specified window. SetCaretPos moves the caret to the specified coordinates. ShowCaret makes the caret visible on the screen at the caret's current position. HideCaret removes the caret from the screen, but does not invalidate the insertion point. DestroyCaret destroys the caret's current shape, frees the caret from the window, and removes the caret from the screen.
(In reply to comment #18) > Will this help? http://msdn.microsoft.com/library/en-us/dntablet/html/carettrack.asp That does help a lot -- I did not realize those methods were actually part of MSAA caret support. Thanks for the info. At least we know what's missing now. I'm sorry to say that unforutunately I don't have any plans currently to expand the MSAA caret support. That's a lot of work and I'm although incredibly busy trying to make Mozilla more accessible, we don't need more caret support for that. Marking HELPWANTED
Keywords: helpwanted
An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi
This bug also appears to affect Mozilla Thunderbird and the Mozilla Suite. The posted XPI extension is a big help... thanks Carter! I applied it to Thunderbird as well, and it works pretty well there with one exception: when inserting the cursor into the body field of a "compose" windows, the TIP button does not appear until at least one keyboard keystroke has been entered. Of course, the idea is to not have to use the keyboard at all. Still, an excellent first realease. I look forward to seeing this integrated into the trunk (please please please). -Greg Mackinnon University of Vermont
Just to clarify, I was not the developer of the extension posted. Credit goes to Jim Hurd. I just stumbled across the link on tabletpcbuzz.com. I did send him an e-mail about this bug though.
How does the extension work?
Jim Hurd's extension does make the floating TIP appear, when the url bar has the focus. But I'm having an additional problem: when using the virtual keyboard mode of the TIP, after a couple keystrokes, the history completion kicks in and interfers with the TIP, making it un-usable.
(In reply to comment #24) > ... the history completion kicks in and interfers with the TIP Can you try Mozilla Seamonkey 1.8a4 and see if it has the same history problem? I recently changed Seamonkey so the history popup didn't fire excess focus events.
Could be related, or could be new bug: Tablet PC Input Panel (TIP) is context-sensitive in MSIE, but not Mozilla 1.7.3 or FF 0.9.3. In MSIE, when cursor is in address bar, these 6 options appear as buttons you can click on to insert text: http:// www. .com .org . / Not so in Mozilla products.
(In reply to comment #20) > An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi Apparently this fix is not compatible with more recent versions. Attempts to install this extension to Firefox 1.0 result in an 'incompatible extension' error message.
This bug still appears in the latest version of Tablet PC with service pack 2 and bug/patches up to 3/22/05
(In reply to comment #27) > (In reply to comment #20) > > An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi > > Apparently this fix is not compatible with more recent versions. Attempts to > install this extension to Firefox 1.0 result in an 'incompatible extension' > error message. See here for how to make it work in later versions: http://www.tabletpcbuzz.com/forum/topic.asp?TOPIC_ID=17827&whichpage=7 However, the tip vanishes after a single keystroke when using the soft keyboard mode. In my opinion, this bug deserves to be bumped to a higher priority. Usability is significantly affected.
*** Bug 309946 has been marked as a duplicate of this bug. ***
(In reply to comment #29) > In my opinion, this bug deserves to be bumped to a higher priority. Usability is > significantly affected. I would heartily agree. The proud new owner of a tablet PC, and Firefox is fairly unusable for me. The hacked extension does do the trick, though.
Depends on: 312941
(In reply to comment #29) > (In reply to comment #27) > See here for how to make it work in later versions: > http://www.tabletpcbuzz.com/forum/topic.asp?TOPIC_ID=17827&whichpage=7 > However, the tip vanishes after a single keystroke when using the soft keyboard > mode. The vanishing-on-autocomplete TIP problem, and TIP awareness is solved by using the correct caret tracking. Very straight-forward to add into widget\src\windows\nsWindow.cpp. I've tried, it works. The best description with example source code is at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winui/winui/windowsuserinterface/resources/carets/usingcarets.asp What then remains to be done is to call SetCaretPosition(x,y). That's the hard part...
(In reply to comment #32) > the correct caret tracking. Very straight-forward to add into > widget\src\windows\nsWindow.cpp. I've tried, it works. You can now download example source code at http://www.hut.fi/~jwagner/firefox/ff-1.5-beta2release-CaretHandlingDemo-bugNr-244119.zip The zip will be up for at least a few months. I must emphasize this is demo/example source code only, NOT a final fix. The code adds TIP/caret support, but there are still issues that need to be ironed out (mainly, SetCaretPos(x,y) correct coordinates==?). See the included readme.txt for details.
*** Bug 317756 has been marked as a duplicate of this bug. ***
The GeckoTIP extension is a fork of the TPCTIP extension mentioned above. It works around Bug 313443 which is the root cause of the "TIP disappears after each keystroke" issue with TPCTIP. It also adds context sensitive URL bar and other features, and works in Firefox/Thunderbird 1.5 as well as 1.0. You can get it at http://geckotip.mozdev.org However, a final fix in the Mozilla codebase is obviously ideal, rather than hacking workarounds...
*** Bug 337691 has been marked as a duplicate of this bug. ***
So will this ever be fixed?
(In reply to comment #37) > So will this ever be fixed? See comment 35. There's an extension that adds TIP support.
QA Contact: os.integration
Latest version of Firefox still has not fixed this problem completely.
Depends on: 313443
*** Bug 348091 has been marked as a duplicate of this bug. ***
*** Bug 346727 has been marked as a duplicate of this bug. ***
Aaron, this is a followup to our discussion in Bug 313443, specifically your comment: > It sounds like this discussion belongs in bug 244119. Anyway, I think it might > be okay to do what you say, but better not in the mozilla/accessible code. It > should be somewhere in nsCaret.cpp or widget/src/windows or someplace that > listens to caret move events. Based on my outline in those comments of a straightforward solution to this bug (244119), and your recommendation above, should the status or assignment of this bug be modified (since it's solution would not involve the Accessibility code)? As I said in the Bug 313443 comments, I'm willing to work with someone familiar with the appropriate Mozilla code to get this bug squashed so Tablet PC and UMPC/Origami users can have native text input support in Mozilla apps. Again, the solution only requires Mozilla to create, move, and destroy an invisible Win32 caret that mirrors the creation, location, and destruction of Mozilla's regular caret. That's it. The OS handles everything else on it's own. In comment 33 Jan Wagner even made the beginnings of a solution, but we clearly need help from someone with more Mozilla codebase experience.
Yes. The discussion of whether it's a good idea or not, and how to code it, and the code itself, belong outside of the accessibility API effort and outside of mozilla/accessible.
Assignee: aaronleventhal → nobody
Just downloaded Firefox on my tablet and the input panel problem still exists. Has anyone found a solution yet?
Is it possible to put a link to the GeckoTIP extension homepage somewhere at the top of the page to avoid repeated comments like the above? For those who don't know, GeckoTIP fixes this bug almost completely. http://geckotip.mozdev.org/ Also, I'm still willing to work with someone on integrating the support into Gecko/Mozilla, but don't know who to approach and I'm not too familiar with the Mozilla codebase. It would involve a few Win32 input caret calls synchronized with the normal cross-platform Mozilla input caret behavior and certain Window change events.
This might be fixed by patch in bug 371273.
This was considered an important bug before! Can someone verify if it's fixed in the recent Firefox 3 builds? http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
Aaron, I tried it out and it doesn't look like it fixed anything related to the floating input panel under Win XP Tablet PC Edition. If I'm reading your comments in bug 371273 correctly, the main change that could be relevant is how the caret location is updated when text is being entered? If so, that's not going to fix this bug (i.e. make Mozilla apps work with the floating TIP w/o GeckoTIP extension), because the OS isn't even recognizing the input fields as input fields to begin with unless GeckoTIP is installed. The only way I know of to make the OS recognize non-windows-native input fields as input fields for the floating TIP is to create the invisible native windows caret as GeckoTIP does. I don't think tweaks to MSAA support will do it by itself. (We did have a related bug whose fix was dependent on proper caret location change events when input is entered-- you already helped us fix that bug-- but even then, it was only in concert with GeckoTIP being installed that we got the floating TIP.) Also, under Vista it seems GeckoTIP no longer works and I'm not sure why yet, so it's even less clear what Mozilla needs to fully support the floating TIP in Vista, if it's even possible without using native input fields. Fortunately Vista's non-floating-TIP is much easier to use so it's much less crucial of a bug under Vista. I'll let you know if I figure out or hear anything new about making Mozilla support the floating TIP under Vista.
I see. Has a bug been filed with Win XP Tablet Edition to get them to recognize MSAA textfields? We're using their API correctly. We shouldn't have to use an invisible caret.
I'm not even sure how one files a Windows bug with MS-- I've looked and can't find a straightforward way to contact them on non-beta software bugs. But I'm guessing since Vista is out they'd be even less interested, and their own MSDN site on the issue recommends using the invisible caret (or native Windows carets to begin with), so it seems they've decided that's how it should work. GeckoTIP seems to work well enough for XP Tablet Edition, but if we want to get Tablet support into Mozilla I think Vista is the OS to shoot for. It has some APIs for directly manipulating the Tablet Input Panel that might be the way to go. I'm still looking into it for the next GeckoTIP version and I'm pretty busy with my own work these days, but again, if someone with more Mozilla experience (not sure what area of Mozilla this falls under) is willing to help maybe we can get the support built in to the Mozilla codebase.
Maybe this isn't too hard then. I think you'd want to patch this code in nsAccessibleWrap::FireAccessibleEvent() if (eventType == nsIAccessibleEvent::EVENT_TEXT_CARET_MOVED) { ... } Currently that's at line 1420: http://lxr.mozilla.org/seamonkey/source/accessible/src/msaa/nsAccessibleWrap.cpp#1420 You'd need to get the bounds of the caret accessible and then set the invisible Windows caret there: nsCOMPtr<nsIAccessible> caretAccessible; rootAccessible->GetCaretAccessible(getter_AddRefs(caretAccessible)); caretAccessible->GetBounds(&x, &y, &width, &height); ::SetCaretPos(x, y); However, it sounds like the coordinates should be relative to the window, which is not a problem. We just need to use GetBounds on the rootAccessible and subtract that.
Attachment #265130 - Flags: review? → review?(emaijala)
Attachment #265129 - Attachment is obsolete: true
Attachment #265129 - Flags: review?
Comment on attachment 265130 [details] [diff] [review] Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen. Three points: * How do I test this with Tablet PC without having one? * I hope it's okay to CreateCaret/SetCaretPos/DestroyCaret all in a row as we do in IME. * I hope it does not interfer with IME which also sets the caret
Aaron, I don't think it's OK to do Create/Set/Destroy in a row-- the floating TIP only works when the caret actually exists, so destroying it should only occur when the Mozilla caret is destroyed (i.e. when a text field loses focus). Similarly the caret should be created when a text field first gains focus. Does this significantly complicate things?
It makes it a little more complicated, yes. But, I it is still doable. What I would do is for each nsCaretAccessible cache the caret visibility status. Then nsIAccessibleCaret should have a method like CheckVisibilityChange(). If visibility changes then we should either create or destroy the caret (not to mention fire an MSAA SHOW/HIDE event for the caret which we're avoiding right now).
Comment on attachment 265130 [details] [diff] [review] Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen. Needs work.
Attachment #265130 - Flags: review?(emaijala)
Attachment #265130 - Attachment is obsolete: true
Ian, do you know if the Windows XP Tablet PC edition activates the a11y code at all in Mozilla? I'm wondering if we can hide the SetCaretPos() code in there. You can tell for sure by installing and using this addon on your Windows XP Tablet PC version of Firefox: https://addons.mozilla.org/en-US/firefox/addon/2407
Aaron, I can confirm that it DOES in Vista (I upgraded recently and don't have an XP Tablet Edition install handy). I'll ask someone with XP Tablet to verify it but I'm guessing it's the same as Vista.
> destroying it should only occur when the Mozilla caret is destroyed (i.e. when > a text field loses focus). The problem occurs in a rich text field, if you arrow from a small character to a larger one (or vice versa). Since the caret must change height, I would have to ::CreateCaret() with the new caret height. But, perhaps if the caret is kept around until the next focus, blur or caret movement, and then can we destroy it, create it and move it all at the same time. (Different from create, move and then quickly destroy which I understand would be a problem).
Aaron, I think destroy, create, move is ok. I checked MS Wordpad in AccEvent32.exe and according to the caret events being logged, that is exactly how their carets are handled, so presumably it should work in Mozilla too. Also, an XP Tablet Edition user has verified to me that, according to the addon you posted before, Win XP Tablet DOES activate the accessibility code.
A fix for this is in bug 381888.
I hate to report but this bug is alive and well in the firefox 3 nightlies when using Windows Vista. The search bar and address bar will have the TIP work a few times if you are starting from a new firefox session with no saved tabs, and the only in the address bar and search bar, but text boxes and other objects within actual webpage content simply do not trigger the TIP at all. Any windows vista user can follow the directions provided here http://ultramobilepc-tips.blogspot.com/2007/02/ok-one-thing-that-i-missed-lot-in-my-q1.html to make their computer think it is a tablet pc for that session, and the setting goes away on reboot. Reproduction can easily be had by making your pc think it is a tablet, or having a tablet, and simply trying to use any Firefox 3 nightly in Windows Vista
So it works more than it used to, but it only works in the UI and not in content?
yes, that is accurate
I will be working on a Google Summer of Code project (Bug 430587) which will fix this bug.
Depends on: 430587
Summary: "In Place Tablet Input Panel" icon doesn't appear in Windows XP service pack 2 (tablet pc), likely because of missing caret tracking support → "In Place Tablet Input Panel" icon doesn't appear for text input
Even though the bug number here is a little lower, I'd like to mark this as a dupe of bug 286072 since that bug applies more broadly to both pen and finger input in win7. If anyone has an issue with that please post back to discuss.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: