1236058 - virtual (on screen) keyboard should not be opened when a bluetooth keyboard is connected

Status: RESOLVED FIXED

(Core :: Widget: Win32, defect)

Description

[(not reading bugmail) Nick Desaulniers [:\n]]

Bug #1007063 now causes the virtual keyboard to be displayed, even when a bluetooth keyboard is connected.  This is frustrating as it wastes screen space.

From a user:

"It's not simply a matter of clicking once to make the virtual keyboard go away. Every time I start Firefox, the on-screen keyboard appears. Every time I open a new browser tab or window, the on-screen keyboard appears. Every time I click in an editable field, the on-screen keyboard appears. Every time Firefox loads a page with focus on a text-input element, the on-screen keyboard appears. The keyboard is always in the way, obscuring my view of the page, like an annoying ad in a pop-up window. There is no way, short of permanently disabling the virtual keyboard, to turn this off."

Comment 1

[:Gijs (he/him)]

Do we have more details about this, or a machine that reproduces? What kind of bluetooth keyboard is this? With what kind of device? Running which version of windows?

A workaround would be toggling the "ui.osk.enabled" preference to false.

However, I'd much rather understand exactly how this is happening and why their keyboard is not being picked up.

Comment 2

[(not reading bugmail) Nick Desaulniers [:\n]]

Windows 8.1 HP tablet hybrid.  Pinging the user for more info...

Comment 3

[:Gijs (he/him)]

(In reply to Nick Desaulniers [:\n] from comment #2)
> Windows 8.1 HP tablet hybrid.  Pinging the user for more info...

That would be useful. Also relevant: when they say "click", do they have a mouse, or a touchpad as part of the bluetooth keyboard - or are they tapping the tablet screen? Is our behaviour different from that of IE or Chrome?

Comment 4

[(not reading bugmail) Nick Desaulniers [:\n]]

They say that the keyboard came with the tablet.

"it shows up in the Device Manager as 'HID Keyboard Device'. It plugs into the tablet to share the power supply, but uses Bluetooth for data."

> Is our behaviour different from that of IE or Chrome?

Sounds like it is.

Comment 5

[(not reading bugmail) Nick Desaulniers [:\n]]

"Sometimes a touchpad that's part of the Bluetooth keyboard, sometimes an external mouse that's plugged into a USB port on the tablet."

Comment 6

[Jared Wein [:jaws] (please needinfo? me)]

We have touch data via bug 1221947, but we still will show the keyboard if the event wasn't a touch. For example, if a mouse clicked on a text field but there was no keyboard, it's not a touch event and there might not be a touch screen, but we want to show the keyboard for the mouse user to be able to type.

Nick, can you ask the person to go to about:config and copy the `ui.osk.debug.keyboardDisplayReason` pref value to share? That will tell us why our code is determining it should show the keyboard.

Comment 7

[:Gijs (he/him)]

(In reply to (Away 12/29-1/3) Jared Wein [:jaws] (please needinfo? me) from comment #6)
> We have touch data via bug 1221947, but we still will show the keyboard if
> the event wasn't a touch. For example, if a mouse clicked on a text field
> but there was no keyboard, it's not a touch event and there might not be a
> touch screen, but we want to show the keyboard for the mouse user to be able
> to type.
> 
> Nick, can you ask the person to go to about:config and copy the
> `ui.osk.debug.keyboardDisplayReason` pref value to share? That will tell us
> why our code is determining it should show the keyboard.

IME the debug information is not amazing when we *do* decide to show the keyboard (when we "shouldn't"). But we'll see, I guess.

(In reply to Nick Desaulniers [:\n] from comment #4)
> They say that the keyboard came with the tablet.

Can they provide the exact model of the tablet? HP has manufactured dozens of them, and "tablet hybrid" could even mean "convertible", ie something that indeed ships with a keyboard, but usually then one into which the tablet "docks", not one that you'd connect over USB specifically... so I'm a little confused.

(In reply to Nick Desaulniers [:\n] from comment #4)
> > Is our behaviour different from that of IE or Chrome?
> 
> Sounds like it is.

Maybe, but our algorithm that determines whether to show the OSK is pretty similar to Chrome's, so it would be useful if we knew for sure if Chrome behaved the same way or not.

Comment 8

[Klemens]

(In reply to :Gijs Kruitbosch from comment #103 from bug 1007063)

> Please can you add exact details about the brand and model "tablet PC"
> you're using, as well as those of the BT keyboard to bug 1236058 ? We would
> love to fix this for everyone, but to do that we need to be able to
> reproduce the issue, which is highly hardware-dependent (for obvious
> reasons, we do hardware detection to determine whether or not a keyboard is
> present etc.).

Happy to provide the info:
Lenovo Thinkpad 8 model no. 20BN002NGE
(Intel Atom Z3770 1.46 GHz, 2 GB RAM, 128 GB SSD)
Lenovo Compact Bluetooth Keyboard KT-1255 with TrackPoint
(model no. is 0B47189 I guess, sold independently, not coming with the tablet, not docked in the tablet in any way, independent/rechargeable battery power)
and in this configuration also using an external screen as an addition to the tablet touch screen
(an old non-touch Lenovo ThinkVision 24" attached via VGA and a Micro-HDMI adapter to the tablet - but this should hardly make any difference)
Software as mentioned before:
Win 8.1 Pro 32-bit with all updates
Firefox 43.0.3 (at that time)

I can confirm that the effect was exactly as described in comment #1 here. Pressing the Esc key on the BT keyboard made the OSK go away (as well as clicking it away with the mouse of course), but any of the triggers mentioned in comment #1 made it pop up again immediately.

My configuration is either tablet in hand, no keyboard or screen connected - pure tablet operation - *or* sitting at a desk with BT keyboard and extra screen like in a traditional PC set-up. In this case, Firefox normally is open on the big screen, and the OSK also appeared there, hugely inflated and untouchable. :-)
(I rarely mix modes, eg touching the screen of the tablet while in desk mode. So it would take extra experiment now to find out about OSK behavior with all the suggested remedies applied)

The OSK bug must have been brought in by one very recent FF update, can't say which - it actually took me a while to realize that something was not arbitrarily but consistently wrong, that simple restarts did not help etc.
My Firefox is on automatic updates, so I cannot tell when exactly this happened. Maybe it was the update that also disabled my old Pixlr Grabber add-on (outdated anyway ;-), or even more likely an update that came shortly before that. For sure I know I had to fill out a lot of online forms all day on Dec 31 and was fighting against OSK pop-ups endlessly. From my notes and docs I am sure the bug was already there on Dec 30 around 4 pm. It might have started pesting me on Dec 29 also around 4 pm. All European time.

Chrome did and does not show anything like it. All other software seemed unchanged, including Windows itself.

This is all I can report I guess. You will surely weed out all the useless info yourself.
Many many thanks for all the good work, guys! Happy to have a non-annoying non-Google non-Microsoft browser back. It makes a difference.

Comment 9

[:Gijs (he/him)]

(In reply to Klemens from comment #8)
> (In reply to :Gijs Kruitbosch from comment #103 from bug 1007063)
> 
> > Please can you add exact details about the brand and model "tablet PC"
> > you're using, as well as those of the BT keyboard to bug 1236058 ? We would
> > love to fix this for everyone, but to do that we need to be able to
> > reproduce the issue, which is highly hardware-dependent (for obvious
> > reasons, we do hardware detection to determine whether or not a keyboard is
> > present etc.).
> 
> Happy to provide the info:

Thanks, this is already really helpful. I'll see if I can get my hands on one of these keyboards, though it seems like Amazon/Lenovo are the only sources. Maybe I can find an equivalent type of device (considering Nick mentioned the user they're in touch with used HP - if it was only HP or only Lenovo devices, it'd be likely the exact brand/model of the device matters, now I'm less sure...) that I can just pick up in a local store, which will be quicker...

> My configuration is either tablet in hand, no keyboard or screen connected -
> pure tablet operation

Just so I know: in this mode (ie when there really isn't a keyboard), is our OSK behaviour correct, in your experience? Or maybe you don't use Firefox / browsers in this mode?

> The OSK bug must have been brought in by one very recent FF update, can't
> say which

We enabled it in Firefox 43, after working out a number of issues. We were only aware of cases where we *don't* pop up the keyboard when we should be (which was basically the status quo anyway), and some edge cases with context menus and/or tabbing to other fields with the OSK where the keyboard disappears (which is annoying, but fixable).

I'm sorry that you're running into the opposite type of bug which is obviously worse.

> Chrome did and does not show anything like it.

The chrome behaviour here is interesting, because the code is very very very similar. I'm not sure offhand what they do differently that would explain this.

One more question: if you go to about:config while the bluetooth keyboard is connected, search for "ui.osk", what is the value of the "ui.osk.debug.keyboardDisplayReason" pref?

Many thanks!

Comment 10

[Klemens]

This starts to be really interesting :-)

Before we do any experiments it might be helpful to consider usage scenarios. I also must try to understand better.

No, I rarely use Firefox in a tablet-only situation. I can not imagine many people doing it. On a tiny screen (like mine) it is sheer pain to fumble in scrollbars, etc. This is true for all of Windows. Bigger screen sizes would not make it much better. I guess whoever uses Windows using touch in desktop mode knows what she/he does. I can only talk about Win 8.x, not about Win 10, but I bet the basic problem has not changed there. If you use touch on a desktop you know you live a bitter compromise. So then, the nuisance of maybe having to call up the OSK from the taskbar (if it does not appear by itself) is in fact the least of all complications. Nevertheless it is a nice move of you to support this by good recognition of OSK. But yes, our opposite type of bug now is much worse.

So, in touch mode you would want touch-focussed software - ie Firefox as a Metro app. Well, we had that (I confess I never got around to even try it out), you gave up on it (I guess with good reason). Thinking about it... I hardly browse in tablet mode at all. If I must do it then I live with desktop Firefox and the said limitations or maybe try IE or Edge for the moment. If there was a Firefox that one could toggle to a touch-friendly mode (meaning wipe instead of scroll bars etc etc, thus better control over OSK decisions) this surely would be worth a lot in that situation!

So rather ask me about PDF readers of which I use two - one for touch, one for desktop. Which is also completely stupid but superiour to any other compromise. So if I would really surf using touch then I bet I would have a different browser prepared for it.

Sorry for the bla bla.
It means the whole predicament is really also the trouble with Windows as a touch OS.
The thought of users having their keyboard attached but folded back must be horror...
But it may also mean that what you had before you decided on improving OSK detection was perhaps already pretty good.

Back to your questions.

What do I see in about:config while typing on said Lenovo keyboard connected to said Lenovo Thinkpad tablet:
ui.osk.debug.keyboardDisplayReason | user set || IKPOS: Lack of keyboard confirmed [all in bold]
ui.osk.enabled | default || true [have not changed this yet...]
ui.osk.on_screen_keyboard_path | user set || - [all in bold, still using the silly workaround, sorry]

Let's go:
Moving the Firefox window over to the tablet touch screen and switching off keyboard. Then tipping on the touch screen somewhat.
This makes ui.osk.debug.keyboardDisplayReason change to "IKPOS: Used touch to focus control [...]"
OSK does not appear when i enter fields on webpages etc. (of course, because path is deliberately broken)
(But actually I don't miss much - has this ever been different? Well, maybe yes. Still...)

Deleting the "-" from ui.osk.on_screen_keyboard_path now.
Ok. The path resets itself to ...\microsoft shared\ink\TabTip.exe
More navigating - OSK appears (on the touch screen, overlapping browser window) whenever field entry is required.
(Nice, ok.)

Switching on BT keyboard again. Moving Firefox window back to big screen.
Now the OSK still appears when I click into random fields. OSK is on touch screen, thus not obstructing browser window (but the location surely may change depending on where the browser window opened originally). This seems somehow familiar to me, maybe the effect is older than I thought - and I did not realize it right away, only when I started working during holidays when I did not have the big screen, only the tablet but with external keyboard. Aha. Firefox 43 came out Dec 15? Could be this was the starting point, yes.

So, setting ui.osk.enabled to false now.
OSK is dead, well done.
Moving Firefox to tablet screen again, switching off BT keyboard.
Ok, OSK is permanently dead. Hm. This is the intention, sure, but is it a terrific solution for my daily life? Hardly.
Especially when I call up the OSK via the taskbar it now appears on the big screen (without touch) for a Firefox window that is on the touch screen...

You know, I have lived with Windows software that correctly and wrongly remembered having been open on the big screen also when there was no big screen attached, so all this poses no real problem. :-)
We could do really funky testing now by unconnecting and reconnecting the screen and see how what this leads to. And moving windows from big to small screen and back during testing... The combinatorial explosion of experiments would be fabulous.

Ok, coming to the hardware configuration.

I would suspect you need not care about this specific type of keyboard...
Any cheap BT keyboard from the store around the corner connected to any Windows tablet might be able to reproduce the behavior. How could a Lenovo keyboard that is supposed to work with all kind of hardware connect in a different way? But you know better about these issues.

Ok, one more experiment. This is really interesting.
BT keyboard turned on. ui.osk.enabled is set back to true again.
I move both Firefox and Chrome windows from the big screen to the small=touch screen.
Now Firefox opens the OSK on the big screen. (This is different from before, strange...)
Chrome also opens an OSK - so their behavior is in fact similar. But the Chrome OSK pops up on the small screen over the Chrome window.
Moving the Firefox window back to the big screen.
Now Firefox opens the OSK on the small screen. (Sort of funny, hm - it always chooses the other place!)
Moving Chrome window back to the big screen.
Chrome does not open OSK at all (which makes sense).

Chrome seems to check on which type of screen it is? If screen is touch, it opens an OSK even though a physical keyboard is connected, but not on non-touch screens? Perhaps. Anyway, different from Firefox.

We might be chasing phantoms here.
If I can try to help more, say a word.

Comment 11

[Klemens]

One more thing :-)
Or two.

1.
Of course Windows has its fingers in here, too.
For example:
OSK that pops up over Firefox window on the big screen can be dragged using mouse to the touch screen and be clicked away.
Next time I enter a text field in the Firefox window OSK pops up again but on the small screen, as anybody would assume.
So by moving windows in the experiment these settings might also shift. I have no idea how much you rely on Windows routines telling you things about the configuration.

2.
Forgot to mention that I benchmarked some other software against Firefox.
Eg not even simple Notepad opens an OSK on the touch screen in desktop mode automatically when my BT is off.
Seems to me a humble stance is totally ok regarding keyboard detection...

Comment 12

[:Gijs (he/him)]

I picked up a Logitech K830, which was the only device I could find in stores close (well, Birmingham is a big place, and I crossed about half of it, but there we are) to me that also has a pointing device (touchpad, in this case), which I figured might be related here. I suspect it's not identifying as a keyboard because it also has the pointing device... With it, I can reproduce the issue on my Surface Pro 3 (obviously with the type cover not connected) when running Windows 10. I'll investigate *why* it's broken as soon as I can, but potentially not today anymore, as it's almost EOD and I'm blocking a few other people on reviews etc. and I should take care of that first.

What is already interesting about comment #10 is:
ui.osk.debug.keyboardDisplayReason | user set || IKPOS: Lack of keyboard confirmed

considering the code here:

https://dxr.mozilla.org/mozilla-central/rev/d4213241bb796fdfa7a5ad4f1989e97b44474364/widget/windows/WinIMEHandler.cpp#740-783

this means that:
(a) all our other checks worked for the tablet (has a touch screen, has rotation/docking sensors, isn't docked, is in use as a tablet rather than a laptop;
(b) there wasn't touch input. Using touch to focus an input field would still cause the OSK to come up in these cases... that is likely not fixable because on some machines the "I have a keyboard but can't use it" and "I have a keyboard and can use it" are not distinguishable. Windows itself shows an OSK in those cases (when configured as such, see below - we might want to tweak what we do on Windows 8 specifically for this case?). But anyway, that likely isn't the majority case here.
(c) Windows must have given us a reasonable response to the request for input devices (or we would have bailed on line 745/746)), but none of them "fit the bill".

Klemens: many thanks for your detailed description. It's quite possible that maybe we want to take slightly different decisions about showing the keyboard on Windows 8. On Windows 10, by default the on-screen keyboard is only in use in tablet mode. There's a preference in the control panel that users can flip, and in that case all apps will start showing the on-screen keyboard. Firefox honours that preference, but AFAIK there isn't such a setting on Windows 8. Also interesting how Chrome deals with this. Not ideal, but still proving to be a "better guesser" than we are, I expect, when in general use.

The first thing I'll try to figure out is why we don't detect the bluetooth keyboard correctly. Everything else we can do separately.

Jared: given the 44 uplift criteria at this time (see the dev-planning list), I don't expect a fix for this issue to be acceptable for uplift to 44. The default Windows 10 behaviour is likely better because of the tablet mode / desktop mode pref. What we *could* easily do (and I think would be upliftable for 44) is flip the default for the require_win10 pref. Given what we know so far, do you think we should do that for 44 ?

Comment 13

[Jared Wein [:jaws] (please needinfo? me)]

Yeah, requiring Windows 10 in the short term is OK with me.

Note that the default value given in all.js for the require_win10 pref is 'false', but the code in WinIMEHandler.cpp has 'true' for the value if the pref isn't found. Changing the default value of the pref to 'true' will match the code in WinIMEHandler.cpp correctly, but I wonder if we should change WinIMEHandler.cpp on mozilla-central?

Comment 14

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #13)
> Yeah, requiring Windows 10 in the short term is OK with me.
> 
> Note that the default value given in all.js for the require_win10 pref is
> 'false', but the code in WinIMEHandler.cpp has 'true' for the value if the
> pref isn't found. Changing the default value of the pref to 'true' will
> match the code in WinIMEHandler.cpp correctly, but I wonder if we should
> change WinIMEHandler.cpp on mozilla-central?

We should, yes. Anywho, I'll disable on win8 for the moment.

One thing I noticed on my win10 machine was that Windows' own apps were equally confused by the bluetooth keyboard...

Comment 15

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to :Gijs Kruitbosch from comment #14)
> One thing I noticed on my win10 machine was that Windows' own apps were
> equally confused by the bluetooth keyboard...

If you can't reproduce on Chrome, maybe you could set up a local Chrome build environment and debug their code? Although it may be harder to determine the reason if their code is simply never entered. The two things that could be happening for them is that maybe they are filtering events better and not getting to the OSK code, or there is some subtlety in their OSK code that isn't present in ours.

Also, we could reach out to their devs and ask if they have heard of similar issues.

Comment 16

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #15)
> (In reply to :Gijs Kruitbosch from comment #14)
> > One thing I noticed on my win10 machine was that Windows' own apps were
> > equally confused by the bluetooth keyboard...
> 
> If you can't reproduce on Chrome, maybe you could set up a local Chrome
> build environment and debug their code? Although it may be harder to
> determine the reason if their code is simply never entered. The two things
> that could be happening for them is that maybe they are filtering events
> better and not getting to the OSK code, or there is some subtlety in their
> OSK code that isn't present in ours.
> 
> Also, we could reach out to their devs and ask if they have heard of similar
> issues.

From some of comment 10 it seems to me like Chrome base part of their decision on whether the screen *on which Chrome is rendered* is a touch screen. Which would explain the difference if the user is using an external screen when using the tablet with a bluetooth keyboard etc. "soft" docking, in a sense, ie not a real "dock" but hooking it up to accessories that essentially let the user use the machine as they would a desktop workstation of some description.

I haven't yet investigated that. First place to look is some of these find device API calls and if we need something else/additional to detect bluetooth devices, or if the item is in the list but we don't detect that it's a keyboard.

Comment 17

[Jared Wein [:jaws] (please needinfo? me)]

There's the fundamental question of if we want to show the OSK for non-touch environments, where someone may be using their mouse to click on the keyboard, or only showing it for touch.

It would be safer to only show it for touch screens, but kiosks(?) would be hindered, though that is a use-case that we don't develop for.

Comment 18

[:Gijs (he/him)]

OK, so my device has an elaborate device_id that starts out with:

HID\{00001812-0000-1000-8000-00805F9B34FB}_DEV_VID&02046D_PID&B335_REV&0010_E693F5C14D49&COL01

I have a patch that fixes this for my local machine and the bluetooth keyboard I bought - but it seems like device IDs are pretty free-form. The  UUID string basically identifies a low-energy bluetooth HID device, but that doesn't otherwise help very much, if not all such devices use this type of device identifier.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff541224%28v=vs.85%29.aspx has no section for HID device identifiers, and the ones generated by HIDClass ( https://msdn.microsoft.com/en-us/library/windows/hardware/ff538842%28v=vs.85%29.aspx ) do not match this type of identifier. I was unable to find more information about this format. :-(

I wrote a patch that keys off _DEV_VID: https://treeherder.mozilla.org/#/jobs?repo=try&revision=39e8c52990c4

and one that keys off the uuid:  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7bea0abe458

I verified that both fix the issue for me on my win10 machine with the bluetooth keyboard I'm testing with.

I'll post some instructions to verify that it fixes it elsewhere once builds are finished, so we know we're not just fixing the one device I happen to be using. If it doesn't, we'll need to get some people to use instructions like http://stackoverflow.com/questions/12567023/how-to-know-usb-device-hid-and-pid to post the hardware identifier of their keyboard.

Jared: I'll try to set up a meeting to discuss what to do about win8/8.1 generally "soon".

Comment 19

[:Gijs (he/him)]

(fwiw, we can't easily use just "HID\" to match because e.g. my surface pro 3 type cover is permanently in the list returned by Windows as HID\MSHW... -- even when it isn't connected.)

Comment 20

[:Gijs (he/him)]

Klemens:

this is where I'd like your help, please. Could you try both of these two builds:

https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-39e8c52990c4b1d81ef004f60227148a4841d611/try-win32/firefox-46.0a1.en-US.win32.zip

https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-b7bea0abe45819b1d45aebdfebc70436d693ed63/try-win32/firefox-46.0a1.en-US.win32.zip

You should be able to unzip each of them in their own folder and run the "firefox.exe" executable you'll find in each. The icons etc. will be like those of Nightly, the development version of Firefox.

Specifically, could you try if either/both of these builds fix the issue for you (ie when the bluetooth keyboard is connected, no on-screen keyboard should show up) and if the behaviour when the bluetooth keyboard isn't connected is still the same (ie showing an on-screen keyboard).

Finally, could you follow the following steps and let me know what shows up:

1) open the "classic" control panel
2) search for "device" (no quotes) in the search box on the top right
3) open the "Device Manager"
4) if you expand "Human Interface Devices" (nb: NOT bluetooth), there should be an entry that represents your keyboard. Mine is called "Bluetooth Low Energy GATT compliant HID device" - I don't know if yours is similar or not, but it is likely that it is. If not, it might have Lenovo or the model number in the title.
5) right click it and select "Properties"
6) switch to the "Details" tab
7) in the dropdown labeled "Property" select "Hardware ID" or "Hardware IDs" and copy the item/items that show up in the "Value" box below (you should be able to right click and select Copy in the context menu to avoid having to type it all out or something).
8) paste that ID / those IDs in a comment here, please. :-)

Basically, we need to establish if the pattern I'm seeing with the device I have is regular, or if every manufacturer just makes stuff up themselves (which is totally possible). :-)

Comment 21

[:Gijs (he/him)]

Vasilica, you did QA on some of the OSK changes. Do you / does your team have access to bluetooth keyboards that you can use to test, and if so can you follow the same set of steps from comment #20 (copied in below) to see what they return for you, please?

(In reply to :Gijs Kruitbosch from comment #20)
> 1) open the "classic" control panel
> 2) search for "device" (no quotes) in the search box on the top right
> 3) open the "Device Manager"
> 4) if you expand "Human Interface Devices" (nb: NOT bluetooth), there should
> be an entry that represents your keyboard. Mine is called "Bluetooth Low
> Energy GATT compliant HID device" - I don't know if yours is similar or not,
> but it is likely that it is. If not, it might have Lenovo or the model
> number in the title.
> 5) right click it and select "Properties"
> 6) switch to the "Details" tab
> 7) in the dropdown labeled "Property" select "Hardware ID" or "Hardware IDs"
> and copy the item/items that show up in the "Value" box below (you should be
> able to right click and select Copy in the context menu to avoid having to
> type it all out or something).
> 8) paste that ID / those IDs in a comment here, please. :-)

Comment 22

[Vasilica Tamas, QA [:vtamas]]

(In reply to :Gijs Kruitbosch from comment #21)
> Vasilica, you did QA on some of the OSK changes. Do you / does your team
> have access to bluetooth keyboards that you can use to test, and if so can
> you follow the same set of steps from comment #20 (copied in below) to see
> what they return for you, please?

Unfortunately we do not have a Bluetooth keyboard. But if is there anything else that I could help you, please tell me.

Comment 23

[Klemens]

(In reply to :Gijs Kruitbosch from comment #20)

> Finally, could you follow the following steps and let me know what shows up:
> 
> 1) open the "classic" control panel
> 2) search for "device" (no quotes) in the search box on the top right
> 3) open the "Device Manager"
> 4) if you expand "Human Interface Devices" (nb: NOT bluetooth), there should
> be an entry that represents your keyboard. Mine is called "Bluetooth Low
> Energy GATT compliant HID device" - I don't know if yours is similar or not,
> but it is likely that it is. If not, it might have Lenovo or the model
> number in the title.
> 5) right click it and select "Properties"
> 6) switch to the "Details" tab
> 7) in the dropdown labeled "Property" select "Hardware ID" or "Hardware IDs"
> and copy the item/items that show up in the "Value" box below (you should be
> able to right click and select Copy in the context menu to avoid having to
> type it all out or something).
> 8) paste that ID / those IDs in a comment here, please. :-)

There are two Hardware ID entries in the value box:

BTHENUM\{00001124-0000-1000-8000-00805f9b34fb}_VID&000217ef_PID&6048
BTHENUM\{00001124-0000-1000-8000-00805f9b34fb}_LOCALMFG&000f

The keyboard btw is registered as "Lenovo BT Interface Device (HID)" with a driver version 1.0.0.9 from 2013-03-27. The driver files have a file version of 0.0.9.9 and a copyright notice "(c) 2013 Chicony" which might mean Lenovo buys there.

Also btw I have two of these keyboards, one in the office and one at home. So I see two of them in the Device Manager list. The Device Manager offers an extra button "Disable" (in the row of buttons underneath the menu bar) only for one of them, so this should be the one that is connected right now.

Nevertheless, the Hardware ID values of both keyboards are identical. Interestingly, also the value for "Is connected" is false for both, and "Is present" is true for both. I don't have to understand the logic here, if there is any. ;-)

Sorry, I was blanketed in work this week, so it took a while to come around. I will try to experiment with your builds asap.

Comment 24

[Klemens]

Ok, I tested both builds now.
I did it in parallel (they were running at the same time), switching BT keyboard off and on and switching ui.osk.enabled in about:config between true and false as needed.

Result: Sorry, I can not see any difference between the builds, and both seem to have the bug.

I started in my usual desktop mode (you know what I mean).
Then when I had moved both browser windows from the external screen to the touch screen, switched ui.osk.enabled on, then switched off the BT keyboard, curiously the OS keyboard did not appear automatically in any fields first (= not from touch interaction!). I could call it up manually from the taskbar though.

It seemed to me that only after I had touched the app symbols in the taskbar the OS keyboard sort of woke up, and from then on both browsers as well loaded the OS keyboard from a touch interaction (nice) as well as from a keyboard/mouse interaction, after I had switched on the BT keyboard again (and this sadly is the bug we are chasing).

The only difference I saw to the rather peculiar behaviour reported first: the OS keyboard always appears on the screen in which the browser windows is located (I have one of the windows open on the touch screen and one on the external screen now)... Oh no, not true, when I move a window to the other screen, its OS keyboard remembers where the window was and appears again on the old screen (where it came up the last time). True for both directions.

What kind of action maybe makes the OS keyboard follow the browser window? Ah, it seems as soon as I switch between browser windows (= builds in this case), the OS keyboard "has learned where it belongs" because then it appears on top of the browser window again. Ok... Yes, this seems to be true whatever I do and whereever I move things.

Maybe some of this helps...

Comment 25

[:Gijs (he/him)]

(In reply to Klemens from comment #24)
> Ok, I tested both builds now.
> I did it in parallel (they were running at the same time), switching BT
> keyboard off and on and switching ui.osk.enabled in about:config between
> true and false as needed.
> 
> Result: Sorry, I can not see any difference between the builds, and both
> seem to have the bug.

Yes, I suspected as much when you commented here with the hardware IDs, but I figured I shouldn't bias your testing... They don't match the patterns I added.


> Maybe some of this helps...

I think the location of the OSK is determined by Windows. We just start the process. I don't know how Windows makes that determination and we can do very little if anything to influence it, I believe. :-(

I'll clear the ni against you.

Jared, based on:

(In reply to Klemens from comment #23)
> BTHENUM\{00001124-0000-1000-8000-00805f9b34fb}_VID&000217ef_PID&6048
> BTHENUM\{00001124-0000-1000-8000-00805f9b34fb}_LOCALMFG&000f

It seems we could either check for "_VID" as a substring, or hardcode both of the bluetooth UUID patterns ({00001124-, and {00001812-). I am tempted to do the former. It seems like the bluetooth spec thinks that even things like glucose meters can be HID devices ( https://developer.bluetooth.org/TechnologyOverview/Pages/HOGP.aspx ). Of course, maybe Windows won't lump them in the keyboard category for devices we request, but I wouldn't want to be holding my breath too much. OTOH, I don't know if the variation between _DEV_VID and _VID isn't indicative of different vendors just all deciding to do what they like here. :-(

It would be nice if we could somehow use prefs or some other vehicle to configure which devices "count" as keyboards so it's easier to update the list, but that would essentially require regexp support on the C++ side and I don't know if that's a possibility.

Comment 26

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Klemens from comment #24)
> Ok, I tested both builds now.
> I did it in parallel (they were running at the same time)

How did you run both builds at the same time? If you didn't specify a separate profile explicitly for them, launching the second executable will just open a new window of the current instance. But it does sound like based on the code changes Gijs made, neither build will fix it for you.

(In reply to :Gijs Kruitbosch from comment #25)
> It seems we could either check for "_VID" as a substring, or hardcode both
> of the bluetooth UUID patterns ({00001124-, and {00001812-). I am tempted to
> do the former. It seems like the bluetooth spec thinks that even things like
> glucose meters can be HID devices (
> https://developer.bluetooth.org/TechnologyOverview/Pages/HOGP.aspx ). Of
> course, maybe Windows won't lump them in the keyboard category for devices
> we request, but I wouldn't want to be holding my breath too much. OTOH, I
> don't know if the variation between _DEV_VID and _VID isn't indicative of
> different vendors just all deciding to do what they like here. :-(

Gijs, we should key off of 0x1124 and 0x1812, as those are both HID devices per the Bluetooth spec. Search for 1124 and 1812 in https://msdn.microsoft.com/en-us/library/windows/apps/dn263090.aspx (even though this page says they are not supported, it appears that they are just not supported for the method of implementation described by that page). _VID appears to just be a generic "Vender ID" substring. I'm OK with treating glucose meters, heart rate monitors, and watches disabling the OSK for now until we learn of a better approach.

Comment 27

[:Gijs (he/him)]

Attachment: MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

Review commit: https://reviewboard.mozilla.org/r/31225/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/31225/

Comment 28

[:Gijs (he/him)]

Comment on attachment 8709004 [details]
MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

f? on bugzilla because mozreview doesn't know about it, but Jared, can you stamp that this is what you had in mind? (I'm fairly sure, but let's be deliberate...)

Comment 29

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(sickish, maybe slower respose)]

Comment on attachment 8709004 [details]
MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

https://reviewboard.mozilla.org/r/31225/#review28103

Otherwise, looks good to me (Although, I'm not sure if the values of the constants are correct).

::: widget/windows/WinIMEHandler.cpp:766
(Diff revision 1)
> +      const std::wstring BT_HID_DEVICE = L"HID\\{00001124";
> +      const std::wstring BT_HOGP_DEVICE = L"HID\\{00001812";

Perhaps, static const?

# Oh, sorry, I reviewed this during business trip, but failed to submit the review result.

Comment 30

[:Gijs (he/him)]

Comment on attachment 8709004 [details]
MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/31225/diff/1-2/

Comment 31

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b4ed16215f73

Comment 32

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/b4ed16215f73

Comment 33

[:Gijs (he/him)]

Comment on attachment 8709004 [details]
MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

Approval Request Comment
[Feature/regressing bug #]: on-screen keyboard support on Windows
[User impact if declined]: on-screen keyboard pops up too much on Windows when using a bluetooth keyboard, which is very annoying
[Describe test coverage new/current, TreeHerder]: because this all depends on hardware, there is no automated test coverage
[Risks and why]: very low - we're basically adding 2 additional "this is probably a keyboard" cases to the code that determines whether or not you have a keyboard attached, to take bluetooth devices into account.
[String/UUID change made/needed]: no

Comment 34

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8709004 [details]
MozReview Request: Bug 1236058 - recognize bluetooth keyboard devices when determining whether to show an on-screen keyboard, r=masayuki,f=jaws

Don't want to annoy more windows users! Taking it.

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-beta/rev/4063364477ac