Closed Bug 1461961 Opened 3 years ago Closed 3 years ago

Support Android Autofill API in GeckoView

Categories

(GeckoView :: General, defect, P2)

defect

Tracking

(firefox61 wontfix, firefox62 affected, firefox63 affected)

RESOLVED DUPLICATE of bug 1330257
Tracking Status
firefox61 --- wontfix
firefox62 --- affected
firefox63 --- affected

People

(Reporter: sebastian, Unassigned)

References

Details

(Whiteboard: [geckoview:fenix][geckoview:klar])

* Developer documentation: https://developer.android.com/preview/features/autofill
* Available from: Android 8.0 (API level 26)

The Autofill API allows third-party apps like password managers to autofill forms in other apps. This is a platform feature and works out of the box when using native Android UI widgets. For web content we would need to implement the API callbacks ourselves.

In GeckoView we'd need to follow this guide:
https://developer.android.com/guide/topics/text/autofill#virtual

Roughly the following needs to happen:
* Once a user focuses a form element, we call notifyEntered() with an id (see below) and a rect so that the autofill popup can show up at the right position:
https://developer.android.com/reference/android/view/autofill/AutofillManager.html#notifyViewEntered(android.view.View,%20int,%20android.graphics.Rect)

* The Android system will call onProvideAutofillVirtualStructure() to get the structure of the form:
https://developer.android.com/reference/android/view/View.html#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
* The WebView documentations contains a good example of how a HTML form structure looks like in code:
https://developer.android.com/reference/android/webkit/WebView.html#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
* We call notifyViewEntered() / notifyViewExited() to notify the framework about focus changes
* We call commit() / cancel() to notify the system whenever the user completed a form or stopped editing it
This will be P1 for fenix, marking P2 for now.
Priority: -- → P2
James, how long it would take to integrate with Autofill API for Android 8.0+? How long for the backported API in Google Play Services that relies on a11y?

Klar+WebView supports the Autofill API, so not supporting in GeckoView would be a feature regression in Klar+GV.
Flags: needinfo?(snorp)
Does the tracking flags means that this bug will be fixed in firefox62?
(In reply to Cheng Zhang from comment #3)
> Does the tracking flags means that this bug will be fixed in firefox62?

Hi Cheng - yes, at least investigated as such, won't be fixed in 61

Tracking flags: 
firefox61 --- wontfix
firefox62 --- affected

"tracking-firefoxXX A flag which show whether a bug is being investigated for possible resolution in the Firefox XX release."

Read more here: https://wiki.mozilla.org/Release_Management/Release_Process
(In reply to Cheng Zhang from comment #3)
> Does the tracking flags means that this bug will be fixed in firefox62?

No. This is far from being fixed (at least in this bug there is no sign of any work on the actual problem at all)
So far it is only clear this won't be fixed in Fx61 and that :cpeterson has investigated this for Fx62.
James is investigating how much work Autofill will take.
I had a look at this over the weekend. The work here seems pretty close to the work required to have a full virtual DOM for a11y, which I know is pretty daunting. The bulk of the work for autofill is in two spots: onProvideAutofillVirtualStructure()[1], and autofill()[2]. The former is for exposing the structure of the web page to the autofill service, and the latter is for actually filling things in with the provided data.

If we can get away with only providing a ViewStructure hierarchy for things identified as login forms, we may be able to get away with something relatively easy here. If we need to have the entire DOM exposed, that's gonna take a while, but I think 2 weeks may be reasonable for the simplified version (if workable). I'm going to NI Eitan here, since he has more insight due to previous a11y work.

The next thing to do may be to wire up a fake view structure and see how various Autofill services respond.

[1] https://developer.android.com/reference/android/view/View#onProvideAutofillVirtualStructure(android.view.ViewStructure,%20int)
[2] https://developer.android.com/reference/android/view/View.html#autofill(android.util.SparseArray%3Candroid.view.autofill.AutofillValue%3E)
Flags: needinfo?(snorp) → needinfo?(eitan)
> If we can get away with only providing a ViewStructure hierarchy for things identified as login forms

Yes, I think it would be enough to just expose a single focused form. I tried that in a prototype some time ago. Also look at the WebView documentation for onProvideAutofillVirtualStructure(). They provide an example how WebView maps a form to the virtual structure.
Unlike a11y, it looks like this API lets you do stuff asynchronously[1]. So we can fill the structure out on-demand, which is awesome. Yeah, probably a minimal structure with some form entry children would be enough for this. Unless there is a visible input[type=password], we can probably return an empty one.

1. https://developer.android.com/reference/android/view/ViewStructure.html#asyncNewChild(int)
Flags: needinfo?(eitan)
(In reply to Sebastian Kaspari (:sebastian) from comment #8)
> > If we can get away with only providing a ViewStructure hierarchy for things identified as login forms
> 
> Yes, I think it would be enough to just expose a single focused form. I
> tried that in a prototype some time ago. Also look at the WebView
> documentation for onProvideAutofillVirtualStructure(). They provide an
> example how WebView maps a form to the virtual structure.

Ah, I hadn't seen that. That does look pretty reasonable.
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) from comment #10)
> (In reply to Sebastian Kaspari (:sebastian) from comment #8)
> > > If we can get away with only providing a ViewStructure hierarchy for things identified as login forms
> > 
> > Yes, I think it would be enough to just expose a single focused form. I
> > tried that in a prototype some time ago. Also look at the WebView
> > documentation for onProvideAutofillVirtualStructure(). They provide an
> > example how WebView maps a form to the virtual structure.
> 
> Ah, I hadn't seen that. That does look pretty reasonable.

Sorry for pinging (I know it can be annoying), but as I'm periodically checking this bug, I wonder if there is any further work to track related to it? This last comment seems pretty optimistic, but nothing was posted since.
The actual implementation seems to have happened in bug 1330257.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1330257
Depends on: 1489569
No longer depends on: 1492704
Product: Firefox for Android → GeckoView
You need to log in before you can comment on or make changes to this bug.