"In Place Tablet Input Panel" icon doesn't appear for text input

RESOLVED DUPLICATE of bug 286072

Status

()

Firefox
Shell Integration
P4
enhancement
RESOLVED DUPLICATE of bug 286072
13 years ago
8 years ago

People

(Reporter: Ryan, Unassigned)

Tracking

({helpwanted})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 obsolete attachments)

(Reporter)

Description

13 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8

Service pack 2 for Windows XP (currently in beta) provides an update to the
pen-input functionality of Windows XP Tablet PC edition.  In supported
applications, an "in place tablet input panel" icon appears/hovers next to any
text input control when the tablet pen hovers over it.  Clicking on the icon
brings up a floating input panel that allows the user to input text directly
into the text box.

This is likely because Firefox doesn't include support for the Microsoft Active
Accessibility API, which is used to track the input caret in text boxes.  For
more information, see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dntablet/html/carettrack.asp

Reproducible: Always
Steps to Reproduce:
1. Install Windows XP SP2 beta on a tablet pc
(http://www.microsoft.com/technet/prodtechnol/winxppro/sp2preview.mspx)
2. Hover the pen over a text input area (e.g., the address bar)

Actual Results:  
No text input panel icon appeared.

Expected Results:  
The input panel icon should have appeared next to the insertion point caret. 
(See, for example, the Internet Explorer address bar after installing the
service pack.)

Comment 1

13 years ago
Can you test with a recent nightly (mozilla or firefox) ? (1.6 is old)
This is probably a issue with Mozilla, not just Firefox
(Reporter)

Comment 2

13 years ago
Tested again with the latest nightly of Mozilla 
(Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a2) Gecko/20040521)

The same issue occurrs.  When the program first launches, the icon shows up once
or twice (clicking on either the address bar or the search box on the default
home   page - http://www.mozilla.org/start/).  A couple of times, I've seen it
appear very briefly and then disappear.  So, it could be that mozilla is
rendering on top of the icon.  Otherwise, it's possible that the text caret
position has an initial default that's never being updated.

Comment 3

13 years ago
This bug happens for me in both firefox and thunderbird

Comment 4

13 years ago
Firefox 0.2.9 displays same behavior under Windows XP Tablet PC Edition Service
Pack 2 Version 2149

Comment 5

13 years ago
(In reply to comment #4)
> Firefox 0.2.9 displays same behavior under Windows XP Tablet PC Edition Service
> Pack 2 Version 2149

Same problem on my Tablet. I'm using Firefox 0.9.2 & Thunderbird 0.7.2

Comment 6

13 years ago
"Firefox  doesn't include support for the Microsoft Active Accessibility API,
which is used to track the input caret in text boxes."

Aaron, is this something you can help us with?
Assignee: bugs → aaronleventhal

Comment 7

13 years ago
(In reply to comment #6)
> "Firefox  doesn't include support for the Microsoft Active Accessibility API,
> which is used to track the input caret in text boxes."
Yes we do have MSAA support in Firefox! Here are the docs:
http://www.mozilla.org/projects/ui/accessibility/vendors-win.html

Let me know if something's not to spec, we have Gecko working with Window-Eyes
which uses MSAA for almost everything.

Comment 8

13 years ago
To verify that we support MSAA, load firefox and accexplore.exe or inspect.exe
from the MSAA SDK tools available on 
http://www.microsoft.com/downloads/details.aspx?FamilyId=4179742F-1F3D-4115-A8BA-2F7A6022B533&displaylang=en

Comment 9

13 years ago
I just thought I'd add that this issue remains in service pack 2 final running
firefox 0.9.3

Comment 10

13 years ago
I've just noticed that the TIP appears correctly on the first page loaded in 
the browser, but stops working as soon as you go to another page. Pressng f7 
to enable/disable caret browsing doesn't help.

Comment 11

13 years ago
(In reply to comment #8)
> To verify that we support MSAA, load firefox and accexplore.exe or inspect.exe
> from the MSAA SDK tools available on 
>
http://www.microsoft.com/downloads/details.aspx?FamilyId=4179742F-1F3D-4115-A8BA-2F7A6022B533&displaylang=en

As other have noted, it works sporadically in Mozilla with the released SP2.
Maybe your MSAA  code is correct and something else is stomping an it?

For the record, here's what Microsoft told me about the problem.

"The TIP uses caret tracking events to determine which text input control has
focus, and can invoke the Input Panel.  The floating TIP icon will not appear
when a caret is not present or if caret events are not being accurately
generated.  (for more information about what is required to enable the floating
TIP icon within an application, see this document:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dntablet/html/carettrack.asp
)"

I'm sure three would be a lot of appreciative tablet users, if someone could
take a look at this problem.

Comment 12

13 years ago
This bug appears in Mozilla 1.7.x and Thunderbird as well, So perhaps it should 
be expanded from just "firefox"?

The inability to use an un-docked TIP in Mozilla is a much bigger inconvenience 
than you might expect. Performing input on only the default top and bottom-
docked TIP can give you serious hand Cramps. I have had to revert to using IE 
and Outlook when using my work Tablet (Gasp!). Perhaps the priority on this bug 
could be bumped up?

Comment 13

13 years ago
I completely agree.  I also have reverted to using IE on my tablet (no worries, 
I still use Firefox on my desktop).  It just makes life much easier for me.

Comment 14

13 years ago
Just tried the pre release 1.0 firefly version . Sorry it's the same. I am 
going to uninstall it and use IE with my tablet PC. It's a shame  that you 
guys don't support tablet PC users any better.

Comment 15

13 years ago
Just tried the pre release 1.0 firefly version . Sorry it's the same. I am 
going to uninstall it and use IE with my tablet PC. It's a shame  that you 
guys don't support tablet PC users any better.

Comment 16

13 years ago
sorry, I  did not mean Firefly (although I really liked the series, shame it 
was cancelled), I was referring to the new Firefox preview release candidate 
1.0. Sorry for posting it twice, was my first post here.

Comment 17

13 years ago
Someone said it works on the first page.

I don't really know what could be wrong. We fire a caret move event. When I run
accevent.exe from the MSAA SDK it reports them, as well as the location of the
caret. It's working beyond the first page.

Can Microsoft point us to a documentation page that describes exactly what calls
they use with the MSAA caret? Maybe that will help.
Priority: -- → P4

Comment 18

13 years ago
Will this help? http://msdn.microsoft.com/library/en-us/dntablet/html/carettrack.asp

To support Input Panel, you must implement the following five functions, which
represent the core caret tracking functionality in MSAA:

CreateCaret creates a new shape for the system caret and assigns ownership of
the caret to the specified window.

SetCaretPos moves the caret to the specified coordinates.

ShowCaret makes the caret visible on the screen at the caret's current position.

HideCaret removes the caret from the screen, but does not invalidate the
insertion point.

DestroyCaret destroys the caret's current shape, frees the caret from the
window, and removes the caret from the screen.

Comment 19

13 years ago
(In reply to comment #18)
> Will this help?
http://msdn.microsoft.com/library/en-us/dntablet/html/carettrack.asp

That does help a lot -- I did not realize those methods were actually part of
MSAA caret support. Thanks for the info.

At least we know what's missing now. I'm sorry to say that unforutunately I
don't have any plans currently to expand the MSAA caret support. That's a lot of
work and I'm although incredibly busy trying to make Mozilla more accessible, we
don't need more caret support for that.

Marking HELPWANTED
Keywords: helpwanted

Comment 20

13 years ago
An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi

Comment 21

13 years ago
This bug also appears to affect Mozilla Thunderbird and the Mozilla Suite.

The posted XPI extension is a big help... thanks Carter!

I applied it to Thunderbird as well, and it works pretty well there with one
exception:  when inserting the cursor into the body field of a "compose"
windows, the TIP button does not appear until at least one keyboard keystroke
has been entered.  Of course, the idea is to not have to use the keyboard at all.

Still, an excellent first realease.  I look forward to seeing this integrated
into the trunk (please please please).

-Greg Mackinnon
University of Vermont

Comment 22

13 years ago
Just to clarify, I was not the developer of the extension posted.  Credit goes
to Jim Hurd.  I just stumbled across the link on tabletpcbuzz.com.  I did send
him an e-mail about this bug though. 

Comment 23

13 years ago
How does the extension work?

Comment 24

13 years ago
Jim Hurd's extension does make the floating TIP appear, when the url bar has the
focus. But I'm having an additional problem: when using the virtual keyboard
mode of the TIP, after a couple keystrokes, the history completion kicks in and
interfers with the TIP, making it un-usable.

Comment 25

13 years ago
(In reply to comment #24)
> ... the history completion kicks in and interfers with the TIP

Can you try Mozilla Seamonkey 1.8a4 and see if it has the same history problem?
I recently changed Seamonkey so the history popup didn't fire excess focus events.

Comment 26

13 years ago
Could be related, or could be new bug:

Tablet PC Input Panel (TIP) is context-sensitive in MSIE, but not Mozilla 1.7.3
or FF 0.9.3.  In MSIE, when cursor is in address bar, these 6 options appear as
buttons you can click on to insert text:

http://
www.
.com
.org
.
/

Not so in Mozilla products.

Comment 27

13 years ago
(In reply to comment #20)
> An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi

Apparently this fix is not compatible with more recent versions.  Attempts to
install this extension to Firefox 1.0 result in an 'incompatible extension'
error message.

Comment 28

12 years ago
This bug still appears in the latest version of Tablet PC with service pack 2
and bug/patches up to 3/22/05

Comment 29

12 years ago
(In reply to comment #27)
> (In reply to comment #20)
> > An extension was developed to fix this at: http://www.datagrove.com/tpctip.xpi
> 
> Apparently this fix is not compatible with more recent versions.  Attempts to
> install this extension to Firefox 1.0 result in an 'incompatible extension'
> error message.

See here for how to make it work in later versions:

http://www.tabletpcbuzz.com/forum/topic.asp?TOPIC_ID=17827&whichpage=7

However, the tip vanishes after a single keystroke when using the soft keyboard
mode.

In my opinion, this bug deserves to be bumped to a higher priority. Usability is
significantly affected.

*** Bug 309946 has been marked as a duplicate of this bug. ***

Comment 31

12 years ago
(In reply to comment #29)

> In my opinion, this bug deserves to be bumped to a higher priority. Usability is
> significantly affected.

I would heartily agree. The proud new owner of a tablet PC, and Firefox is
fairly unusable for me. The hacked extension does do the trick, though.

Updated

12 years ago
Depends on: 312941

Comment 32

12 years ago
(In reply to comment #29)
> (In reply to comment #27)
> See here for how to make it work in later versions:
> http://www.tabletpcbuzz.com/forum/topic.asp?TOPIC_ID=17827&whichpage=7
> However, the tip vanishes after a single keystroke when using the soft keyboard
> mode.

The vanishing-on-autocomplete TIP problem, and TIP awareness is solved by using the correct caret tracking. Very straight-forward to add into widget\src\windows\nsWindow.cpp. I've tried, it works.

The best description with example source code is at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winui/winui/windowsuserinterface/resources/carets/usingcarets.asp

What then remains to be done is to call SetCaretPosition(x,y). That's the hard part...

Comment 33

12 years ago
(In reply to comment #32)
> the correct caret tracking. Very straight-forward to add into
> widget\src\windows\nsWindow.cpp. I've tried, it works.

You can now download example source code at
http://www.hut.fi/~jwagner/firefox/ff-1.5-beta2release-CaretHandlingDemo-bugNr-244119.zip
The zip will be up for at least a few months.

I must emphasize this is demo/example source code only, NOT a final fix. The code adds TIP/caret support, but there are still issues that need to be ironed out (mainly, SetCaretPos(x,y) correct coordinates==?). See the included readme.txt for details.

Comment 34

12 years ago
*** Bug 317756 has been marked as a duplicate of this bug. ***

Comment 35

12 years ago
The GeckoTIP extension is a fork of the TPCTIP extension mentioned above. It works around Bug 313443 which is the root cause of the "TIP disappears after each keystroke" issue with TPCTIP. It also adds context sensitive URL bar and other features, and works in Firefox/Thunderbird 1.5 as well as 1.0. You can get it at 
http://geckotip.mozdev.org

However, a final fix in the Mozilla codebase is obviously ideal, rather than hacking workarounds...

Comment 36

11 years ago
*** Bug 337691 has been marked as a duplicate of this bug. ***

Comment 37

11 years ago
So will this ever be fixed?

Comment 38

11 years ago
(In reply to comment #37)
> So will this ever be fixed?

See comment 35. There's an extension that adds TIP support.

QA Contact: os.integration

Comment 39

11 years ago
Latest version of Firefox still has not fixed this problem completely.

Updated

11 years ago
Depends on: 313443
*** Bug 348091 has been marked as a duplicate of this bug. ***
*** Bug 346727 has been marked as a duplicate of this bug. ***

Comment 42

11 years ago
Aaron, this is a followup to our discussion in Bug 313443, specifically your comment:

> It sounds like this discussion belongs in bug 244119. Anyway, I think it might 
> be okay to do what you say, but better not in the mozilla/accessible code. It 
> should be somewhere in nsCaret.cpp or widget/src/windows or someplace that 
> listens to caret move events.

Based on my outline in those comments of a straightforward solution to this bug (244119), and your recommendation above, should the status or assignment of this bug be modified (since it's solution would not involve the Accessibility code)? As I said in the Bug 313443 comments, I'm willing to work with someone familiar with the appropriate Mozilla code to get this bug squashed so Tablet PC and UMPC/Origami users can have native text input support in Mozilla apps. Again, the solution only requires Mozilla to create, move, and destroy an invisible Win32 caret that mirrors the creation, location, and destruction of Mozilla's regular caret. That's it. The OS handles everything else on it's own. In comment 33 Jan Wagner even made the beginnings of a solution, but we clearly need help from someone with more Mozilla codebase experience.

Comment 43

11 years ago
Yes. The discussion of whether it's a good idea or not, and how to code it, and the code itself, belong outside of the accessibility API effort and outside of mozilla/accessible.
Assignee: aaronleventhal → nobody

Comment 44

10 years ago
Just downloaded Firefox on my tablet and the input panel problem still exists. Has anyone found a solution yet?

Comment 45

10 years ago
Is it possible to put a link to the GeckoTIP extension homepage somewhere at the top of the page to avoid repeated comments like the above? For those who don't know, GeckoTIP fixes this bug almost completely.
http://geckotip.mozdev.org/

Also, I'm still willing to work with someone on integrating the support into Gecko/Mozilla, but don't know who to approach and I'm not too familiar with the Mozilla codebase. It would involve a few Win32 input caret calls synchronized with the normal cross-platform Mozilla input caret behavior and certain Window change events.

Comment 46

10 years ago
This might be fixed by patch in bug 371273.

Comment 47

10 years ago
This was considered an important bug before!

Can someone verify if it's fixed in the recent Firefox 3 builds?
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

Comment 48

10 years ago
Aaron, I tried it out and it doesn't look like it fixed anything related to the floating input panel under Win XP Tablet PC Edition. If I'm reading your comments in bug 371273 correctly, the main change that could be relevant is how the caret location is updated when text is being entered? If so, that's not going to fix this bug (i.e. make Mozilla apps work with the floating TIP w/o GeckoTIP extension), because the OS isn't even recognizing the input fields as input fields to begin with unless GeckoTIP is installed. The only way I know of to make the OS recognize non-windows-native input fields as input fields for the floating TIP is to create the invisible native windows caret as GeckoTIP does. I don't think tweaks to MSAA support will do it by itself. (We did have a related bug whose fix was dependent on proper caret location change events when input is entered-- you already helped us fix that bug-- but even then, it was only in concert with GeckoTIP being installed that we got the floating TIP.) 

Also, under Vista it seems GeckoTIP no longer works and I'm not sure why yet, so it's even less clear what Mozilla needs to fully support the floating TIP in Vista, if it's even possible without using native input fields. Fortunately Vista's non-floating-TIP is much easier to use so it's much less crucial of a bug under Vista. I'll let you know if I figure out or hear anything new about making Mozilla support the floating TIP under Vista.

Comment 49

10 years ago
I see. Has a bug been filed with Win XP Tablet Edition to get them to recognize MSAA textfields? We're using their API correctly. We shouldn't have to use an invisible caret.

Comment 50

10 years ago
I'm not even sure how one files a Windows bug with MS-- I've looked and can't find a straightforward way to contact them on non-beta software bugs. But I'm guessing since Vista is out they'd be even less interested, and their own MSDN site on the issue recommends using the invisible caret (or native Windows carets to begin with), so it seems they've decided that's how it should work.

GeckoTIP seems to work well enough for XP Tablet Edition, but if we want to get Tablet support into Mozilla I think Vista is the OS to shoot for. It has some APIs for directly manipulating the Tablet Input Panel that might be the way to go. I'm still looking into it for the next GeckoTIP version and I'm pretty busy with my own work these days, but again, if someone with more Mozilla experience (not sure what area of Mozilla this falls under) is willing to help maybe we can get the support built in to the Mozilla codebase. 

Comment 51

10 years ago
Maybe this isn't too hard then.

I think you'd want to patch this code in nsAccessibleWrap::FireAccessibleEvent()
if (eventType == nsIAccessibleEvent::EVENT_TEXT_CARET_MOVED) {
  ...
}
Currently that's at line 1420:
http://lxr.mozilla.org/seamonkey/source/accessible/src/msaa/nsAccessibleWrap.cpp#1420

You'd need to get the bounds of the caret accessible and then set the invisible Windows caret there:
nsCOMPtr<nsIAccessible> caretAccessible;
rootAccessible->GetCaretAccessible(getter_AddRefs(caretAccessible));
caretAccessible->GetBounds(&x, &y, &width, &height);
::SetCaretPos(x, y);

However, it sounds like the coordinates should be relative to the window, which is not a problem. We just need to use GetBounds on the rootAccessible and subtract that.

Comment 52

10 years ago
Created attachment 265129 [details] [diff] [review]
Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen.
Attachment #265129 - Flags: review?

Comment 53

10 years ago
Created attachment 265130 [details] [diff] [review]
Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen.
Attachment #265130 - Flags: review?

Updated

10 years ago
Attachment #265130 - Flags: review? → review?(emaijala)

Updated

10 years ago
Attachment #265129 - Attachment is obsolete: true
Attachment #265129 - Flags: review?

Comment 54

10 years ago
Comment on attachment 265130 [details] [diff] [review]
Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen.

Three points:
* How do I test this with Tablet PC without having one?
* I hope it's okay to CreateCaret/SetCaretPos/DestroyCaret all in a row as we do in IME.
* I hope it does not interfer with IME which also sets the caret

Comment 55

10 years ago
Aaron, I don't think it's OK to do Create/Set/Destroy in a row-- the floating TIP only works when the caret actually exists, so destroying it should only occur when the Mozilla caret is destroyed (i.e. when a text field loses focus). Similarly the caret should be created when a text field first gains focus. Does this significantly complicate things? 

Comment 56

10 years ago
It makes it a little more complicated, yes. But, I it is still doable.

What I would do is for each nsCaretAccessible cache the caret visibility status. Then nsIAccessibleCaret should have a method like CheckVisibilityChange(). If visibility changes then we should either create or destroy the caret (not to mention fire an MSAA SHOW/HIDE event for the caret which we're avoiding right now).

Comment 57

10 years ago
Comment on attachment 265130 [details] [diff] [review]
Move invisible system caret when Mozilla caret moves. Accessibility must be active for it to happen.

Needs work.
Attachment #265130 - Flags: review?(emaijala)

Updated

10 years ago
Attachment #265130 - Attachment is obsolete: true

Comment 58

10 years ago
Ian, do you know if the Windows XP Tablet PC edition activates the a11y code at all in Mozilla?

I'm wondering if we can hide the SetCaretPos() code in there.

You can tell for sure by installing and using this addon on your Windows XP Tablet PC version of Firefox:
https://addons.mozilla.org/en-US/firefox/addon/2407

Comment 59

10 years ago
Aaron, I can confirm that it DOES in Vista (I upgraded recently and don't have an XP Tablet Edition install handy). I'll ask someone with XP Tablet to verify it but I'm guessing it's the same as Vista.

Comment 60

10 years ago
> destroying it should only occur when the Mozilla caret is destroyed (i.e. when 
> a text field loses focus).

The problem occurs in a rich text field, if you arrow from a small character to a larger one (or vice versa). Since the caret must change height, I would have to ::CreateCaret() with the new caret height.

But, perhaps if the caret is kept around until the next focus, blur or caret movement, and then can we destroy it, create it and move it all at the same time. (Different from create, move and then quickly destroy which I understand would be a problem).

Comment 61

10 years ago
Aaron, I think destroy, create, move is ok. I checked MS Wordpad in AccEvent32.exe and according to the caret events being logged, that is exactly how their carets are handled, so presumably it should work in Mozilla too. 

Also, an XP Tablet Edition user has verified to me that, according to the addon you posted before, Win XP Tablet DOES activate the accessibility code.

Comment 62

10 years ago
A fix for this is in bug 381888.

Comment 63

10 years ago
I hate to report but this bug is alive and well in the firefox 3 nightlies when using Windows Vista. The search bar and address bar will have the TIP work a few times if you are starting from a new firefox session with no saved tabs, and the only in the address bar and search bar, but text boxes and other objects within actual webpage content simply do not trigger the TIP at all. 

Any windows vista user can follow the directions provided here http://ultramobilepc-tips.blogspot.com/2007/02/ok-one-thing-that-i-missed-lot-in-my-q1.html to make their computer think it is a tablet pc for that session, and the setting goes away on reboot. Reproduction can easily be had by making your pc think it is a tablet, or having a tablet, and simply trying to use any Firefox 3 nightly in Windows Vista

Comment 64

10 years ago
So it works more than it used to, but it only works in the UI and not in content?

Comment 65

10 years ago
yes, that is accurate
Duplicate of this bug: 417464
I will be working on a Google Summer of Code project (Bug 430587) which will fix this bug.
Depends on: 430587
Depends on: 88831
Duplicate of this bug: 427692

Updated

8 years ago
Summary: "In Place Tablet Input Panel" icon doesn't appear in Windows XP service pack 2 (tablet pc), likely because of missing caret tracking support → "In Place Tablet Input Panel" icon doesn't appear for text input

Comment 69

8 years ago
Even though the bug number here is a little lower, I'd like to mark this as a dupe of bug 286072 since that bug applies more broadly to both pen and finger input in win7. If anyone has an issue with that please post back to discuss.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 286072
You need to log in before you can comment on or make changes to this bug.