1575401 - [meta] GV support for login autofill

Status: RESOLVED FIXED

(GeckoView :: General, enhancement, P2)

Description

[Emily Kager [:ekager]]

Fenix wants to be able to autofill passwords from fxa into login fields (and save fields) without users needing to install another app.

We investigated becoming an autofill service provider like LockWise does but think this won't work for 2 reasons.

1. The Android Autofill API is only on API Versions O+
2. If we were an autofill service provider, we would have to make the user register Fenix as their lone autofill service, which don't want to make users do as it will break all other apps that need autofill.

For Fenix to be able to autofill itself without autofilling other apps (as Lockwise does) and for autofilling below Oreo, we think we will need hooks into login fields.

This could possibly be done through a webextension, I am the first to admit I am not an expert here, but we should start the discussion as we prioritize bugs for Fennec -> Fenix.

NI James for his expertise to make sure this makes sense

Comment 1

[Vlad Filippov (:vladikoff)]

This could possibly be done through a webextension, I am the first to admit I am not an expert here, but we should start the discussion as we prioritize bugs for Fennec -> Fenix.

Yes, without knowing anything about Geckoview parts..... what we CAN do is build a webextension and borrow the magical browser.js from Fennec, which does form detection for usernames and passwords and do autofill that way.

Comment 2

[Emily Kager [:ekager]]

To clarify, this is separate of supporting the system autofill APIs which GV already does.

We think that ideally we'd want to hook into the way Gecko identifies login forms and then delegate to us for operations like get and save username/password pairs.

Adding Emily to see if this makes sense to their team.

Comment 3

[Chris Peterson [:cpeterson]]

Fenix meta issue for login management: https://github.com/mozilla-mobile/fenix/issues/1284

Comment 4

[Chris Peterson [:cpeterson]]

WONTFIX because snorp says GV already has the necessary API: GV's SessionTextInput API. For Android O+ you would still probably want to use a system autofill provider. Perhaps A-C can create an autofill abstraction for Fenix that encapsulates both GV's SessionTextInput API and Oreo's autofill API, depending on the OS version at run time?

Comment 5

[Matthew N. [:MattN]]

I agree with Emily that using a system autofill provider to fill web forms in Fenix isn't going to work well for the reasons she outlined and it won't work nearly as well as the Toolkit password manager that was used in Fennec.

(In reply to Vlad Filippov (:vladikoff) from comment #1)

Yes, without knowing anything about Geckoview parts..... what we CAN do is build a webextension and borrow the magical browser.js from Fennec, which does form detection for usernames and passwords and do autofill that way.

Please do not fork the toolkit login hueristic code like Fx iOS did… we still haven't unforked it and iOS has missed out on various improvements over the last few years as a result.

Ideally this would be properly supported in GeckoView like it was in Fennec, sharing the toolkit code.

(In reply to Chris Peterson [:cpeterson] from comment #3)

Fenix meta issue for login management: https://github.com/mozilla-mobile/fenix/issues/1284

Management and autofill are very different problems btw.

(In reply to Chris Peterson [:cpeterson] from comment #4)

WONTFIX because snorp says GV already has the necessary API: GV's SessionTextInput API. For Android O+ you would still probably want to use a system autofill provider. Perhaps A-C can create an autofill abstraction for Fenix that encapsulates both GV's SessionTextInput API and Oreo's autofill API, depending on the OS version at run time?

Having to re-implement all of the toolkit password manager heuristics in Fenix seems like the wrong approach to me. Exposing the existing toolkit password manager would give much better results.

This issue was about fill but the equally important part of the equation is capture and I don't believe GV has support for that yet… i.e. saving password from registration or change forms. A browser password manager that only fills but doesn't save isn't very helpful.

Comment 6

[Sebastian Kaspari (:sebastian; :pocmo)]

I disagree with the WONTFIX.

I would prefer us to re-use the code we already have in Gecko to detect login forms etc. and would love to see us provide a high-level API to feed it with data.

(In reply to Chris Peterson [:cpeterson] from comment #4)

Perhaps A-C can create an autofill abstraction for Fenix that encapsulates both GV's SessionTextInput API and Oreo's autofill API, depending on the OS version at run time?

This will be a massive effort. You are asking us to hook into the autofill APIs and basically re-implement the autofill service of Android - including the autofill provider. Also it a pretty inconvenient process: We'd have to use the autofill APIs of GeckoView to ask for the form structure currently visible and then do our own heuristics to find login forms, then use the autofill APIs to fill those forms. Just building those heuristics is not something small and we'd have to maintain all that. We'd have to build a big part of what Lockwise does and mock the Autofill system that Android provides.

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

I believe GeckoView already detects usernames and password fields for the APIs we expose today. This detection[0] is done using the code Matt is referring to in comment #5. The fields are exposed via SessionTextInput.onProvideAutofillStructure()[1], which fills a ViewStructure with the field information. The username and password fields can be identified via the various hints set on the ViewStructure. One problem here is that ViewStructure is only in API 23, so we would need to solve that somehow.

This will be a massive effort. You are asking us to hook into the autofill APIs and basically re-implement the autofill service of Android - including the autofill provider.

The bulk of an autofill service is storing the credentials. Since that's the stated goal, it doesn't seem avoidable.

Also it a pretty inconvenient process: We'd have to use the autofill APIs of GeckoView to ask for the form structure currently visible and then do our own heuristics to find login forms, then use the autofill APIs to fill those forms. Just building those heuristics is not something small and we'd have to maintain all that. We'd have to build a big part of what Lockwise does and mock the Autofill system that Android provides.

I don't believe there will be any complicated heuristics required, because GeckoView already marks username/password fields appropriately.

I agree it's a big project -- you do need to reimplement a lot of what Lockwise is doing. But I don't see how that's avoidable if you want Fenix to provide the same services as Lockwise.

[0] https://searchfox.org/mozilla-central/source/mobile/android/modules/geckoview/GeckoViewAutoFill.jsm#155
[1] https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/SessionTextInput.java#549(In
[2] https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/SessionTextInput.java#713

Comment 8

[Sebastian Kaspari (:sebastian; :pocmo)]

The DefaultDelegate seems to deal with a bunch of things we do not want to mess with. Would it make sense to at least extract the autofill pieces to a separate delegate that we can implement without having to deal with any of the other things that seem to require an implementation?

I still wonder if we couldn't come up with a better API then us listening to the autofill events, then asking for the structure, then going through that structure and then autofilling/saving. Just breaking that up in an "autofill"-only delegate with high-level methods for the events would help a lot, I think.

The bulk of an autofill service is storing the credentials. Since that's the stated goal, it doesn't seem avoidable.

There's just a bunch of indirection here if we implement something that acts as the system autofill provider and an autofill service.

I agree it's a big project -- you do need to reimplement a lot of what Lockwise is doing. But I don't see how that's avoidable if you want Fenix to provide the same services as Lockwise.

Fenix is not going to provide an autofill service to the system / other apps. So I think our scope is much smaller (filling logins). So I was hoping for a less "generic" autofill API that was meant to autofill any Android app with anything.

Comment 9

[Emily Kager [:ekager]]

After our meeting we filed the initial bugs we will need from GV so clearing the NI requests. Thanks!

Comment 10

[Chris Peterson [:cpeterson]]

Reopening this bug as the meta bug for GV work to support login autofill.

Comment 11

[Chris Peterson [:cpeterson]]

Emily suggests login autofill be a top priority for October ([geckoview:m1910]).