1461961 - Support Android Autofill API in GeckoView

Status: RESOLVED DUPLICATE of bug 1330257

(GeckoView :: General, defect, P2)

Description

[Sebastian Kaspari (:sebastian; :pocmo)]

* Developer documentation: https://developer.android.com/preview/features/autofill
* Available from: Android 8.0 (API level 26)

The Autofill API allows third-party apps like password managers to autofill forms in other apps. This is a platform feature and works out of the box when using native Android UI widgets. For web content we would need to implement the API callbacks ourselves.

In GeckoView we'd need to follow this guide:
https://developer.android.com/guide/topics/text/autofill#virtual

Roughly the following needs to happen:
* Once a user focuses a form element, we call notifyEntered() with an id (see below) and a rect so that the autofill popup can show up at the right position:
https://developer.android.com/reference/android/view/autofill/AutofillManager.html#notifyViewEntered(android.view.View,%20int,%20android.graphics.Rect)

* The Android system will call onProvideAutofillVirtualStructure() to get the structure of the form:
https://developer.android.com/reference/android/view/View.html#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
* The WebView documentations contains a good example of how a HTML form structure looks like in code:
https://developer.android.com/reference/android/webkit/WebView.html#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
* We call notifyViewEntered() / notifyViewExited() to notify the framework about focus changes
* We call commit() / cancel() to notify the system whenever the user completed a form or stopped editing it

Comment 1

[David Bolter [:davidb] (NeedInfo me for attention)]

This will be P1 for fenix, marking P2 for now.

Comment 2

[Chris Peterson [:cpeterson]]

James, how long it would take to integrate with Autofill API for Android 8.0+? How long for the backported API in Google Play Services that relies on a11y?

Klar+WebView supports the Autofill API, so not supporting in GeckoView would be a feature regression in Klar+GV.

Comment 3

[Cheng Zhang]

Does the tracking flags means that this bug will be fixed in firefox62?

Comment 4

[brian+github]

(In reply to Cheng Zhang from comment #3)
> Does the tracking flags means that this bug will be fixed in firefox62?

Hi Cheng - yes, at least investigated as such, won't be fixed in 61

Tracking flags: 
firefox61 --- wontfix
firefox62 --- affected

"tracking-firefoxXX A flag which show whether a bug is being investigated for possible resolution in the Firefox XX release."

Read more here: https://wiki.mozilla.org/Release_Management/Release_Process

Comment 5

[Mastacheata]

(In reply to Cheng Zhang from comment #3)
> Does the tracking flags means that this bug will be fixed in firefox62?

No. This is far from being fixed (at least in this bug there is no sign of any work on the actual problem at all)
So far it is only clear this won't be fixed in Fx61 and that :cpeterson has investigated this for Fx62.

Comment 6

[Chris Peterson [:cpeterson]]

James is investigating how much work Autofill will take.

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

I had a look at this over the weekend. The work here seems pretty close to the work required to have a full virtual DOM for a11y, which I know is pretty daunting. The bulk of the work for autofill is in two spots: onProvideAutofillVirtualStructure()[1], and autofill()[2]. The former is for exposing the structure of the web page to the autofill service, and the latter is for actually filling things in with the provided data.

If we can get away with only providing a ViewStructure hierarchy for things identified as login forms, we may be able to get away with something relatively easy here. If we need to have the entire DOM exposed, that's gonna take a while, but I think 2 weeks may be reasonable for the simplified version (if workable). I'm going to NI Eitan here, since he has more insight due to previous a11y work.

The next thing to do may be to wire up a fake view structure and see how various Autofill services respond.

[1] https://developer.android.com/reference/android/view/View#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
[2] https://developer.android.com/reference/android/view/View.html#autofill(android.util.SparseArray%3Candroid.view.autofill.AutofillValue%3E)

Comment 8

[Sebastian Kaspari (:sebastian; :pocmo)]

> If we can get away with only providing a ViewStructure hierarchy for things identified as login forms

Yes, I think it would be enough to just expose a single focused form. I tried that in a prototype some time ago. Also look at the WebView documentation for onProvideAutofillVirtualStructure(). They provide an example how WebView maps a form to the virtual structure.

Comment 9

[Eitan Isaacson [:eeejay]]

Unlike a11y, it looks like this API lets you do stuff asynchronously[1]. So we can fill the structure out on-demand, which is awesome. Yeah, probably a minimal structure with some form entry children would be enough for this. Unless there is a visible input[type=password], we can probably return an empty one.

1. https://developer.android.com/reference/android/view/ViewStructure.html#asyncNewChild(int)

Comment 10

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Sebastian Kaspari (:sebastian) from comment #8)
> > If we can get away with only providing a ViewStructure hierarchy for things identified as login forms
> 
> Yes, I think it would be enough to just expose a single focused form. I
> tried that in a prototype some time ago. Also look at the WebView
> documentation for onProvideAutofillVirtualStructure(). They provide an
> example how WebView maps a form to the virtual structure.

Ah, I hadn't seen that. That does look pretty reasonable.

Comment 11

[Ingvar Stepanyan (RReverser)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #10)
> (In reply to Sebastian Kaspari (:sebastian) from comment #8)
> > > If we can get away with only providing a ViewStructure hierarchy for things identified as login forms
> > 
> > Yes, I think it would be enough to just expose a single focused form. I
> > tried that in a prototype some time ago. Also look at the WebView
> > documentation for onProvideAutofillVirtualStructure(). They provide an
> > example how WebView maps a form to the virtual structure.
> 
> Ah, I hadn't seen that. That does look pretty reasonable.

Sorry for pinging (I know it can be annoying), but as I'm periodically checking this bug, I wonder if there is any further work to track related to it? This last comment seems pretty optimistic, but nothing was posted since.

Comment 12

[Sebastian Kaspari (:sebastian; :pocmo)]

The actual implementation seems to have happened in bug 1330257.