789038 - On screen keyboard doesn't popup in Linux

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P2)

Description

[Jan Horak [:jhorak]]

While on-screen keyboard shows up every time when user select input box by mouse for all GTK application, it doesn't work for Firefox or Thunderbird under Linux.

This feature use caribou GTK module (https://live.gnome.org/Caribou) which connects to focus events. Caribou then fire dbus request for showing onscreen keyboard. It seems that Firefox's input box doesn't trigger GTK focus event.

Original report: https://bugzilla.redhat.com/show_bug.cgi?id=746112

Comment 1

[Mounir Lamouri (:mounir)]

What should take care of that? The DOM FocusManager, the a11y FocusManager or something in widget/?

Comment 2

[Karl Tomlinson (:karlt)]

In order to trigger the Caribou GTK module we'd have to give focus to a GtkEditable or GtkTextView.  I don't know whether we could do that without realizing the widget and it would be very awkward then redirecting keyboard events back to Gecko.

It would be much better to use the DBus API directly.  Is that the toolkit-independent API?

How does Google Chrome do this?

The other part of the solution is getting events from input elements.  Perhaps a11y might be the appropriate solution there.  How does mobile get notifications?

Comment 3

[Jan Horak [:jhorak]]

It sends dbus events like:
dest=org.gnome.Caribou.Keyboard serial=31 path=/org/gnome/Caribou/Keyboard; interface=org.gnome.Caribou.Keyboard; member=Show

dest=org.gnome.Caribou.Keyboard serial=35 path=/org/gnome/Caribou/Keyboard; interface=org.gnome.Caribou.Keyboard; member=Hide

These dbus events creates caribou-gtk module itself so creating same events in gecko may not be general solution for this problem, just workaround.

Comment 4

[alexander :surkov (:asurkov)]

(In reply to Karl Tomlinson (:karlt) from comment #2)

> The other part of the solution is getting events from input elements. 
> Perhaps a11y might be the appropriate solution there.  How does mobile get
> notifications?

I believe a11y fires a11y focus event when input is focused (and a11y is enabled of course) but GTK focus event should be a different thing.

Comment 5

[Morgan Read]

New since 2012 - nothing since 2012
Firefox needs this fix to work on a gnome tablet...
Gnome tablets are much more common than in 2012 - I didn't have one in 2012, but I do now
Firefox tends to be the default web-browser under gnome
A fix for this is urgently needed for an increasing common firefox use scenario

Comment 6

[luciano]

Can anybody reproduce bug #1231067 ? The user is implying that by enabling the on-screen keyboard in gnome accessibility settings, the keyboard _does_ appear, although it interferes with the auto-suggestions of the url bar.

Personally switching it on doesn't make a difference for me. I'm on gnome 3.20 and firefox 49.

Comment 7

[Jehan]

GNOME 3.20.2 (Fedora 24)
Firefox 49.0

I confirm the on-screen keyboard does **not** appear in Firefox, for me neither, which is a serious lack of functionality when it will be used on a tablet.

Also tested from a self-built nightly (with code from the repository, about 2 weeks old) and does not work either.

Comment 8

[Koala Yeung]

The issue still exists in Firefox 50.1.0 on Ubuntu GNOME 16.10

Comment 9

[Koala Yeung]

After further test, I can't seems to switch tab by selecting with touch. That is when I touch the tab with my finger, the touched tab did not activate like when I clicked it with mouse cursor.

I also failed to put caret on the URL bar by touch.

Using Firefox with "MOZ_USE_XINPUT2=1" in env.

Comment 10

[Jan Horak [:jhorak]]

Situation on Fedora 26, caribou and Firefox Nightly:
1. keyboard does not appear at all. When switching to the Firefox window I can see Hide request for caribou keyboard in dbus-monitor (I'm not completely sure if that request comes from Firefox, but when focusing from application ).
 or 
2. keyboard popups when input element is focused but it does not hide when input has no longer focus (like clicking on the body of the page. The keyboard only disappear when switching to different app. When returning back to Firefox the keyboard does not show up until input element is focused. 

Btw caribou basically listen for at-spi events which are dispatched by atk library. So it has nothing to do with Input Method and calls of gtk_im_context_focus_in (as I've wrongly assumed before). There seems to be problem with ignoring blur event which should call:
 atk_object_notify_state_change(atkObj, ATK_STATE_FOCUSED, false);
That is what webkit does there: https://github.com/WebKit/webkit/blob/master/Source/WebCore/accessibility/atk/AXObjectCacheAtk.cpp#L342
(they keep oldObject to remove focus of the previous item before adding it to the new item, we probably won't go that way).

It would be nice to bring a11y guys into this discussion, Karl, any hint who to add?

Comment 11

[Karl Tomlinson (:karlt)]

That sounds promising.  I don't have any further suggestions to add, but let's move this to a11y to for visibility to the right people.  I suspect they are already CCed, but this will have better visibility in the right component.

Comment 12

[Jan Horak [:jhorak]]

It seems that I've moved a little bit further. I'm afraid we're hitting some timeout of dbus request (it's 800ms) when caribou is trying to get caret offset by a11y bus and therefore the Firefox is then added to the hung_processes list there [1] and all subsequent tries of getting caret offset fails because of it [2].

That would explain why the keyboard popups for many times but it stop appearing eventually (I' unable to get some reproduction steps) until Firefox is restarted.

I don't know how to fix this, looks like atk_* calls are made from main thread and it can become busy eventually. I'm not sure if we can move atk_calls to another thread (how much effort it requires).

Backtrace why the caret offset is requested:

#0  check_for_hang (message=message@entry=, bus=, bus_name=":1.74", error=<optimized out>) at atspi-misc.c:1050
#1  _atspi_dbus_get_property (obj=<optimized out>, interface=<optimized out>, name=<optimized out>, name@entry="CaretOffset", error=, type=type@entry="i", data=data@entry=) at atspi-m
isc.c:1240
#2  atspi_text_get_caret_offset (obj=<optimized out>, error=<optimized out>) at atspi-text.c:135
#3  caribou_daemon_set_entry_location (error=, acc=0x559cd89915d0 [AtspiAccessible], self=0x559cd0a90e20 [CaribouDaemon]) at daemon.c:780
#4  caribou_daemon_on_focus (error=, event=<optimized out>, self=0x559cd0a90e20 [CaribouDaemon]) at daemon.c:984

[1] https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1051
[2] https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1038

Comment 13

[Jan Horak [:jhorak]]

For the reference, atspi_set_timeout (where the 800ms comes from):
https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1625

Comment 14

[Mike Gorse]

Attachment: AT-SPI patch to poll direct D-Bus connections

Is anyone able to test this patch to at-spi2-core? If a process is flagged as hung, then libatspi sends a ping and is supposed to unmark the process as hung once a reply is received, but we weren't setting up the direct connections to be polled in the main loop.
I think that this would be a harmless fix, but I'm leary about touching anything involving the main loop. I've tested briefly with orca (lowering the timeout in order to make it easier to reproduce) and was no longer able to reproduce the bug, but it wouldn't hurt if someone else could also test it.

Comment 15

[Makoto Kato [:m_kato]]

Mike, thank you for patch.

But next, please create a patch via mercurial.  See document of https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/index.html

Comment 16

[Makoto Kato [:m_kato]]

Comment on attachment 8874034 [details] [diff] [review]
AT-SPI patch to poll direct D-Bus connections

Karl, could you review this?  (although patch format and C++ format aren't valid,  I will rebase it)

Comment 17

[Mike Gorse]

Fwiw, it is a patch to at-spi2-core, not firefox. I was trying to ask if anyone could test that it fixes the problem described in comment 12.

Comment 18

[Jan Horak [:jhorak]]

(In reply to Mike Gorse from comment #17)
> Fwiw, it is a patch to at-spi2-core, not firefox. I was trying to ask if
> anyone could test that it fixes the problem described in comment 12.
Thanks that you looked into it!
Looks like the patch fixed the issue, at least remove_hung_process is called on caribou stack, which never happened before during my debugging session.

With patch applied I've not encounter losing popup keyboard yet.

Comment 19

[Mike Gorse]

I've pushed the at-spi2-core patch to master: c475b5
Will be in 2.25.3.

Comment 20

[mario.limonciello]

I backported c475b5 onto at-spi2-core on Ubuntu 17.10 (FF 54), launched firefox with MOZ_USE_XINPUT2=1 but still am not seeing the gnome OSK popup on any text fields or address bar.  Are there other pieces missing?
OSK does pop up for other GTK apps like gnome-terminal.

Comment 21

[hendrik.steinhorst]

May I ask if there are any updates on this topic? I desperately try to run firefox on Gnome with a touchscreen device and need the on screen keyboard. Thank you in advance.

Comment 22

[Botond Ballo [:botond]]

I'm not aware of anyone actively working on this. I'm cc'ing a couple of people who are involved with / may potentially be interested in GTK widget enhancements.

Comment 23

[hendrik.steinhorst]

just to be sure, is this treated as an enhancement? From my perspective this is a crucial feature that is not working. My device is currently unusable without the possibility of using an on screen keyboard in firefox on a touch device.

Comment 24

[Jan Horak [:jhorak]]

I've just tried the Firefox with Wayland backend it it seems to be working fairly well under Gnome (Fedora). Please try that with:

$ export MOZ_ENABLE_WAYLAND=1 
$ firefox

Comment 25

[hendrik.steinhorst]

This seems to work, thank you very much