Closed Bug 789038 Opened 12 years ago Closed 3 years ago

On screen keyboard doesn't popup in Linux

Categories

(Core :: Disability Access APIs, defect, P2)

Product:
Core
Component:
Disability Access APIs
Platform:
x86_64
Linux
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: jhorak, Unassigned)

References

Details

(Whiteboard: tpi:+)

Attachments

(1 file)

AT-SPI patch to poll direct D-Bus connections
1.82 KB, patch
Details | Diff | Splinter Review 
While on-screen keyboard shows up every time when user select input box by mouse for all GTK application, it doesn't work for Firefox or Thunderbird under Linux.

This feature use caribou GTK module (https://live.gnome.org/Caribou) which connects to focus events. Caribou then fire dbus request for showing onscreen keyboard. It seems that Firefox's input box doesn't trigger GTK focus event.

Original report: https://bugzilla.redhat.com/show_bug.cgi?id=746112
What should take care of that? The DOM FocusManager, the a11y FocusManager or something in widget/?
In order to trigger the Caribou GTK module we'd have to give focus to a GtkEditable or GtkTextView.  I don't know whether we could do that without realizing the widget and it would be very awkward then redirecting keyboard events back to Gecko.

It would be much better to use the DBus API directly.  Is that the toolkit-independent API?

How does Google Chrome do this?

The other part of the solution is getting events from input elements.  Perhaps a11y might be the appropriate solution there.  How does mobile get notifications?
It sends dbus events like:
dest=org.gnome.Caribou.Keyboard serial=31 path=/org/gnome/Caribou/Keyboard; interface=org.gnome.Caribou.Keyboard; member=Show

dest=org.gnome.Caribou.Keyboard serial=35 path=/org/gnome/Caribou/Keyboard; interface=org.gnome.Caribou.Keyboard; member=Hide

These dbus events creates caribou-gtk module itself so creating same events in gecko may not be general solution for this problem, just workaround.
(In reply to Karl Tomlinson (:karlt) from comment #2)

> The other part of the solution is getting events from input elements. 
> Perhaps a11y might be the appropriate solution there.  How does mobile get
> notifications?

I believe a11y fires a11y focus event when input is focused (and a11y is enabled of course) but GTK focus event should be a different thing.
New since 2012 - nothing since 2012
Firefox needs this fix to work on a gnome tablet...
Gnome tablets are much more common than in 2012 - I didn't have one in 2012, but I do now
Firefox tends to be the default web-browser under gnome
A fix for this is urgently needed for an increasing common firefox use scenario
Priority: -- → P1
Whiteboard: tpi:+
Can anybody reproduce bug #1231067 ? The user is implying that by enabling the on-screen keyboard in gnome accessibility settings, the keyboard _does_ appear, although it interferes with the auto-suggestions of the url bar.

Personally switching it on doesn't make a difference for me. I'm on gnome 3.20 and firefox 49.
GNOME 3.20.2 (Fedora 24)
Firefox 49.0

I confirm the on-screen keyboard does **not** appear in Firefox, for me neither, which is a serious lack of functionality when it will be used on a tablet.

Also tested from a self-built nightly (with code from the repository, about 2 weeks old) and does not work either.
After further test, I can't seems to switch tab by selecting with touch. That is when I touch the tab with my finger, the touched tab did not activate like when I clicked it with mouse cursor.

I also failed to put caret on the URL bar by touch.

Using Firefox with "MOZ_USE_XINPUT2=1" in env.
Situation on Fedora 26, caribou and Firefox Nightly:
1. keyboard does not appear at all. When switching to the Firefox window I can see Hide request for caribou keyboard in dbus-monitor (I'm not completely sure if that request comes from Firefox, but when focusing from application ).
 or 
2. keyboard popups when input element is focused but it does not hide when input has no longer focus (like clicking on the body of the page. The keyboard only disappear when switching to different app. When returning back to Firefox the keyboard does not show up until input element is focused. 

Btw caribou basically listen for at-spi events which are dispatched by atk library. So it has nothing to do with Input Method and calls of gtk_im_context_focus_in (as I've wrongly assumed before). There seems to be problem with ignoring blur event which should call:
 atk_object_notify_state_change(atkObj, ATK_STATE_FOCUSED, false);
That is what webkit does there: https://github.com/WebKit/webkit/blob/master/Source/WebCore/accessibility/atk/AXObjectCacheAtk.cpp#L342
(they keep oldObject to remove focus of the previous item before adding it to the new item, we probably won't go that way).

It would be nice to bring a11y guys into this discussion, Karl, any hint who to add?
Flags: needinfo?(karlt)
That sounds promising.  I don't have any further suggestions to add, but let's move this to a11y to for visibility to the right people.  I suspect they are already CCed, but this will have better visibility in the right component.
Component: Widget: Gtk → Disability Access APIs
Flags: needinfo?(karlt)
It seems that I've moved a little bit further. I'm afraid we're hitting some timeout of dbus request (it's 800ms) when caribou is trying to get caret offset by a11y bus and therefore the Firefox is then added to the hung_processes list there [1] and all subsequent tries of getting caret offset fails because of it [2].

That would explain why the keyboard popups for many times but it stop appearing eventually (I' unable to get some reproduction steps) until Firefox is restarted.

I don't know how to fix this, looks like atk_* calls are made from main thread and it can become busy eventually. I'm not sure if we can move atk_calls to another thread (how much effort it requires).

Backtrace why the caret offset is requested:

#0  check_for_hang (message=message@entry=, bus=, bus_name=":1.74", error=<optimized out>) at atspi-misc.c:1050
#1  _atspi_dbus_get_property (obj=<optimized out>, interface=<optimized out>, name=<optimized out>, name@entry="CaretOffset", error=, type=type@entry="i", data=data@entry=) at atspi-m
isc.c:1240
#2  atspi_text_get_caret_offset (obj=<optimized out>, error=<optimized out>) at atspi-text.c:135
#3  caribou_daemon_set_entry_location (error=, acc=0x559cd89915d0 [AtspiAccessible], self=0x559cd0a90e20 [CaribouDaemon]) at daemon.c:780
#4  caribou_daemon_on_focus (error=, event=<optimized out>, self=0x559cd0a90e20 [CaribouDaemon]) at daemon.c:984

[1] https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1051
[2] https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1038
For the reference, atspi_set_timeout (where the 800ms comes from):
https://github.com/GNOME/at-spi2-core/blob/master/atspi/atspi-misc.c#L1625
Is anyone able to test this patch to at-spi2-core? If a process is flagged as hung, then libatspi sends a ping and is supposed to unmark the process as hung once a reply is received, but we weren't setting up the direct connections to be polled in the main loop.
I think that this would be a harmless fix, but I'm leary about touching anything involving the main loop. I've tested briefly with orca (lowering the timeout in order to make it easier to reproduce) and was no longer able to reproduce the bug, but it wouldn't hurt if someone else could also test it.
Comment on attachment 8874034 [details] [diff] [review]
AT-SPI patch to poll direct D-Bus connections

Karl, could you review this?  (although patch format and C++ format aren't valid,  I will rebase it)
Attachment #8874034 - Flags: review?(karlt)
Fwiw, it is a patch to at-spi2-core, not firefox. I was trying to ask if anyone could test that it fixes the problem described in comment 12.
Attachment #8874034 - Flags: review?(karlt)
(In reply to Mike Gorse from comment #17)
> Fwiw, it is a patch to at-spi2-core, not firefox. I was trying to ask if
> anyone could test that it fixes the problem described in comment 12.
Thanks that you looked into it!
Looks like the patch fixed the issue, at least remove_hung_process is called on caribou stack, which never happened before during my debugging session.

With patch applied I've not encounter losing popup keyboard yet.
I've pushed the at-spi2-core patch to master: c475b5
Will be in 2.25.3.
I backported c475b5 onto at-spi2-core on Ubuntu 17.10 (FF 54), launched firefox with MOZ_USE_XINPUT2=1 but still am not seeing the gnome OSK popup on any text fields or address bar.  Are there other pieces missing?
OSK does pop up for other GTK apps like gnome-terminal.
Priority: P1 → P2
See Also: → 1370496

May I ask if there are any updates on this topic? I desperately try to run firefox on Gnome with a touchscreen device and need the on screen keyboard. Thank you in advance.

I'm not aware of anyone actively working on this. I'm cc'ing a couple of people who are involved with / may potentially be interested in GTK widget enhancements.

just to be sure, is this treated as an enhancement? From my perspective this is a crucial feature that is not working. My device is currently unusable without the possibility of using an on screen keyboard in firefox on a touch device.

I've just tried the Firefox with Wayland backend it it seems to be working fairly well under Gnome (Fedora). Please try that with:

$ export MOZ_ENABLE_WAYLAND=1 
$ firefox

This seems to work, thank you very much

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: