1343317 - Investigate making GeckoView accessible on UiAutomator (DOM Tree visible)

Status: RESOLVED WONTFIX

(GeckoView :: General, defect, P2)

Description

[No-Jun Park [:njpark]]

With a lot of backlog on Fennec for Android side ( https://trello.com/b/a47BZyic/android-browser-core), now would be a good time to revisit the automation testing approach.

Google’s UiAutomatorviewer app shows the DOM tree for the Fennec, and for the id=layer_view, it has 3 GeckoView subviews. If we can make GeckoView display its sub-views, then Appium/UiAutomator/Espresso test framework would be able to see and interact with those elements as well as the chrome view.  My understanding is that since GeckoView is a kind of custom view, if we can implement the required methods, it should be able to expose its subviews.  

The necessary tasks would include: 
Handle directional controller clicks
Implement accessibility API methods
Send AccessibilityEvent objects specific to your custom view
Populate AccessibilityEvent and AccessibilityNodeInfo for your view
For more detail, see https://developer.android.com/guide/topics/ui/accessibility/custom-views.html

For this Bug, all we want to know is whether it is actually possible, and how much time would be needed to do it. A Fennec developer familiar with GeckoView implementation would be able to assess accurately.

Comment 1

[No-Jun Park [:njpark]]

If this is possible, then we can use Google or Amazon's device farm to run tests on the nightly builds.

Comment 2

[Chris Peterson [:cpeterson]]

Now that Focus/Klar is switching from Chromium WebView to GeckoView, their UI tests are blocked on this bug.

Comment 3

[Chris Peterson [:cpeterson]]

P3 because James says this will be a lot of work. We need to figure out what Klar's UI tests are specifically trying to do.

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

Eitan I'm guessing you can own this for next steps.

Comment 5

[Eitan Isaacson [:eeejay]]

Yup. I'll look into this.

Comment 6

[Eitan Isaacson [:eeejay]]

UIAutomator does use Android's accessibility API.

https://android.googlesource.com/platform/frameworks/uiautomator/+/master/src/com/android/uiautomator/core/UiDevice.java#760

For this to work, we will indeed need a proper accessibility hierarchy in geckoview. We are currently debating our a11y support approach. The fact that we will want uiautomator support puts more emphasis on a full hierarchy implementation.

Comment 7

[David Bolter [:davidb] (NeedInfo me for attention)]

Accessibility support in GeckoView is a blocker and considered P1. We need to morph this bug or create a new bug that is more clearly understandable as Accessibility support blocks GV ship.

Comment 8

[Eitan Isaacson [:eeejay]]

Created an a11y api metabug.