1465155 - On-screen keyboard not working with Click2Speak

Status: NEW

(Core :: Widget: Win32, defect, P3)

Description

[Liz Henry (:lizzard) (relman/hg->git project)]

We have a report that the on-screen keyboard doesn't work in Firefox with https://www.click2speak.net/ with Windows PCs and tablets.

From email: "Since Quantum update, our users can't use our integrated prediction engine on Firefox. It seems that FF input fields can't handle well our typing speed, so it cause skipping of some of the chars and the user gets incomplete words. "

Maybe an event handling issue?

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

Andrei, can your team test this in nightly 62 and get good STR?

Comment 2

[James Teh [:Jamie]]

While this is certainly an accessibility tool, it's unclear as to how accessibility APIs are being used here (if they're being used at all). They're probably just using Win32 APIs to "type" text, in which case this isn't related to accessibility APIs. (That's not to say it isn't a bug; it's just a bug that belongs in a different component.) Further information has been requested from the reporter.

Comment 3

[uriel.katz]

Hi All,

My name is Uriel Katz and I am one of the developers of Click2Speak.

We emulate the keyboard trough Win32 API as that is the only (as far as we know) universal way to produce keyboard strokes that mimic as closely as possible the actual keyboard.

We use SendInput to send the keystrokes, when we want to enter text (say from our prediction engine) we do a call to SendInput per char with a specific delay between each call of 30ms, when typing on Firefox (since Quantum) few of the keystrokes seem to not be registered so for example click on the prediction "happy" will be typed as "hppy".

When we change the delay between calls to around 150-200ms it works but the typing is quite slow and this is only required in Firefox Quantum, we haven't experienced this issue in other programs.

Maybe there is a problem with the event loop in firefox? is it using the normal Win32 message loop?

Comment 4

[Marco Zehe (:MarcoZ)]

This is something for our Win32 widget colleagues. Redirecting the bug there. Jim, who could take this?

Comment 5

[Cristian Baica [:cbaica], Release Desktop QA]

Hi everyone,

We tried to reproduce this issue on PC, a Windows Surface Pro tablet and a Dell XPS 13 with touchpad in order to get specific STR and were successfull in doing so on the Surface Pro tablet and the Dell XPS 13, using Fx 62.0a1, Fx 61.0b9 and Fx61.0b10.

This issue occurs when the user tries to input suggestions in textareas (e.g. when writing an email). The issue does NOT occur when suggestions are written in the address bar.

STR:
1. Launch Click2Speak keyboard and Firefox.
2. Go to gmail.com or outlook.live.com and login.
3. Compose a new email.
4. In the Email text area, start writing some text.
5. Use the suggestions received on the keyboard for different words.

One of the things we noticed is that on the Surface Pro tablet the keyboard is slower in writing the words selected from the suggestion section than on PC.

Comment 6

[Cristian Baica [:cbaica], Release Desktop QA]

Attachment: click2speak-keyboard issues

Comment 7

[Jim Mathies [:jimm]]

(In reply to uriel.katz from comment #3)
> Hi All,
> 
> My name is Uriel Katz and I am one of the developers of Click2Speak.
> 
> We emulate the keyboard trough Win32 API as that is the only (as far as we
> know) universal way to produce keyboard strokes that mimic as closely as
> possible the actual keyboard.
> 
> We use SendInput to send the keystrokes, when we want to enter text (say
> from our prediction engine) we do a call to SendInput per char with a
> specific delay between each call of 30ms, when typing on Firefox (since
> Quantum) few of the keystrokes seem to not be registered so for example
> click on the prediction "happy" will be typed as "hppy".
> 
> When we change the delay between calls to around 150-200ms it works but the
> typing is quite slow and this is only required in Firefox Quantum, we
> haven't experienced this issue in other programs.
> 
> Maybe there is a problem with the event loop in firefox? is it using the
> normal Win32 message loop?

Yes, we use a normal win32 event loop. I'm not sure why SendInput fails for you we use that as well in our automated testing.

https://searchfox.org/mozilla-central/source/widget/windows/nsAppShell.cpp

Comment 8

[uriel.katz]

I think the problem lays with the speed of your loop, it doesn't fail if we send messages (via SendInput) at 100ms intervals when we move to 30ms (which is around 33 messages per second) it "misses" few of them.

This didn't happen before Quantum or in other programs so it must be something with your event loop.