Closed Bug 1003877 Opened 6 years ago Closed 5 years ago

[meta] Custom server support for Firefox Accounts on Android

Categories

(Firefox for Android :: Android Sync, defect)

All
Android
defect
Not set

Tracking

()

RESOLVED FIXED
Tracking Status
fennec 33+ ---

People

(Reporter: nalexander, Assigned: nalexander)

References

Details

(Keywords: meta, Whiteboard: [qa+])

This is the Android counterpart to Bug 989756.  This ticket tracks implementing support for entering custom URLs and propagating them through the entire FxAccount system.

My first thought is that we mirror a set of about:config Gecko preferences to background services and use those from within FxAccount*Activity* to create new FxAccounts.  If we can avoid needing Gecko preferences at Sync time, our lives would be simpler.
This should be tracking-33, since it is part of the great Old Sync Deprecation Story.
See Also: → 989756
tracking-fennec: --- → 33+
Whiteboard: [qa+]
I've been rolling this around in my head overnight.  There are bunch of pieces here, and the technical ones are not the hard ones.

Technically, I anticipate the mirroring approach I advocate above to work just fine.  We will add prefs observers to a small list of background services related prefs, and write them to Android SharedPrefs when they change.  There will be some co-ordination required around default values, but it's all manageable.

My first suggestion is that we only use these pref values at creation time.  That is, the prefs are read at Android Account creation time, cached in the created Account object, and immutable from then on.  There will be sadness on all kinds of levels if Account parameters can shift over time.  (It's possible that other parameters will change, like log-levels.)

I am not interested in ever exposing UI that would let you configure these Account endpoints.  This is the 1% of the 1% that care about doing this.  about:config is sufficient.

I am concerned that we will need to message users that they are using a "non-standard" FxA + Sync configuration.  At best, this will be confusing, because doing the same steps will not configure the same Account; and at worst, it will be easy to phish a user's credentials by changing their FxA auth server endpoint.  I suggest that for users in non-standard configurations, we surface the auth-server URL somewhere close to the email address in sign up, sign in, and re-authenticate.

Supporting multiple end-point URLs will also make it harder to pin our TLS sessions, should we choose to do so.  (We don't currently pin our TLS sessions.)

Finally, this might be the correct time to have /account/create and /account/login direct the client to the token server to talk to.  (The token server already directs the client to the sync endpoint.)  This would be a "minimal discovery" protocol.
Assignee: nobody → nalexander
callahad: a list of desktop prefs that you think are relevant to self-serve users would be awesome.  Thanks!
Flags: needinfo?(dan.callahan)
Also, let me know what I can do to help. I am trying to make May my custom/DIY server month...
The extremely bare-bones "run your own fxa" document notes the necessary prefs on desktop:

  https://docs.services.mozilla.com/howtos/run-fxa.html

In addition to the tokenServerURI for sync itself, the following are necessary to use self-hosted FxA on desktop:

  """
  use your auth-server URL to replace “api.accounts.firefox.com” in the following settings:
      identity.fxaccounts.auth.uri
  use your content-server URL to replace “accounts.firefox.com” in the following settings:
      identity.fxaccounts.remote.uri
      identity.fxaccounts.remote.force_auth.uri
      identity.fxaccounts.settings.uri
  """

Some of these are "jelly" URLs which may not be applicable on android.

FWIW, I would be happy to have "self-hosted sync with mozilla-hosted FxA" as a first goal, meaning only tokenServerURI is editable.  This seems to raise a lot fewer complications than changing the FxA servers, which is a right thorny mess on desktop as well even though it's technically possible.
Flags: needinfo?(dan.callahan)
> FWIW, I would be happy to have "self-hosted sync with mozilla-hosted FxA"

To which I'll add:  the sweet-spot may be mozilla-hosted FxA, self-hosted sync storage, and class C encryption.  If we can enable that via some config fiddling (e.g. you configure tokenServerURI and manually input a strong encryption key) then the resulting system will be pretty nice, even if the setup process is something only an uber-tech-privacy-nerd could tolerate.

This would be a very solid start and IMHO substantially better than the current self-hosting situation for new sync.
Here's a workaround, in case anyone feels as desperate as I did.

I spent hours trying to set up my own account server, gave up on that, and then spent the rest of the day getting the sync server up and running (*cough* I made some typos...), only to discover...this bug. Not wanting all that work to be for naught, I found a way to change this setting on Android without recompiling, though it's a tiny bit messy.

For anyone who wants to use their own sync server in Firefox for Android now (but not auth server), consider trying this. I've only tried it with beta 30.
There's probably a way to streamline this process, maybe even to eliminate airplane mode, but this is what worked for me. I didn't try anything else.

Summary: Firefox stores the sync URIs as user data for your sync account. They can be edited in the Android accounts sqlite database (/data/system/users/?/accounts.db), in the extras table. This probably needs to be done before authentication data has been populated.

Prerequisites: root access, an sqlite editor (aShell + aSQLiteManager do the job), a quick way into airplane mode (or to toggle off network/Wi-Fi access), a file browser that can access /data/

WARNING: These are potentially dangerous operations! You could damage your device... Warranty warranty, mileage may vary, rabble rabble...
1) Begin to set up your sync account using the Accounts interface rather than Firefox itself. Enter airplane mode as quickly as possible after tapping to sign in. Don't even wait to see it finish signing in. Make sure it actually signed in correctly, with Firefox showing the typical response with the green checkmark (I think). If it didn't, try again but not quite so fast.
2) Make sure it didn't get *too* far in the process. Browse to /data/data/org.mozilla[...]/files and ensure fxa.account.json is NOT present. If it is, try starting over and see if you can get into airplane mode more quickly.
3) Open /data/system/users/0/accounts.db. (With aShell + aSQLiteManager, this means opening aShell with Root checked and browsing to the file.)
4) Open the accounts table and note the ID for your Firefox account.
5) Open the extras table, and go to the end of the table. (With aSQLiteManager, this means hit Menu + Last.)
6) Edit the "audience" and "tokenServerURI" entries which correspond to the ID of your Firefox account. (There shouldn't be more than one of each, anyway...) Change them to your sync server URIs, e.g. https://my.sync.server:5001 and https://my.sync.server:5001/token/1.0/sync/1.5
7) Reboot. This step may not be necessary, but it was a tricky process so I didn't want to risk it failing from not rebooting.
8) Turn off airplane mode, and see if you can sync. If all goes well, it will work, and if you happen to check out /data/data/org.mozilla[...]/files/fxa.account.json, it should also reflect your proper sync URI.
(In reply to tw-bugs-firefox from comment #7)
> Here's a workaround, in case anyone feels as desperate as I did.

I sincerely applaud your effort, and *especially* your willingness to help others do the same.  This is, of course, very delicate, and moderately likely to break in future; but this will absolutely work.

For your own edification: if fxa.account.json exists, and an Android Firefox Account does not exist, the details in fxa.account.json will be used to re-create the account (the next time something FxA-related happens).  (Don't ask: it's a work-around for Android deleting accounts when Firefox is stored on the SD card.)  Deleting the Android account yourself should clean up any existing fxa.account.json, but you can always put it back in place yourself.

So I think you can do this much more easily.  In fact, using the aboved and the fact that *debug* Nightlies are android:debuggable, you can edit fxa.account.json without root!

To prevent this being necessary, I'm going to take this ticket.  Again, I applaud your creativity.
Status: NEW → ASSIGNED
Hey Nick, any word on this? Any way I can help out?
(In reply to Dan Callahan [:callahad] from comment #9)
> Hey Nick, any word on this? Any way I can help out?

This is blocked on non-technological discussions: are we going to support custom servers as a feature?  I assumed that we were, and started prototyping how best to support this (see [1]).  After discussion with rfeeley, ckarlof, and others, it's not clear to me that we're going to invest any engineering time into supporting this on any platform, let alone making it a feature of the product.  In response, ckarlof initiated a discussion about doing this in an add-on, and I tried to steer that conversation [2] to a general "statement that we care about custom servers".  If we do care about this feature -- and I certainly care! -- then we can evaluate technology.  The last thing I want is to implement a one-off set of "custom server URLs" that are never tested and then break as Fennec's technology evolves.

You can help by adding your voice to the dev-fxacct discussion, and helping define what our level of commitment to custom server configurations really is.

[1] I've just pushed the android-sync side of one prototype to https://github.com/mozilla-services/android-sync/tree/nalexander/bug-1003877-choose-where-to-sync/.  I haven't pushed the Fennec glue yet.

[2] https://mail.mozilla.org/pipermail/dev-fxacct/2014-June/000911.html
Thanks for the update, Nick...  Some grand words of support, there.  (I now realize that the remark about dev-fxacct probably was only directed at developers, so although I've successfully subscribed to it, I'll remain a mouse, there.)

Even if Mozilla decides not to document or invest extra effort in custom sync server support or software, if at least a small amount could be directed toward an about:config option for configuring custom sync server parameters on Android, it'd be appreciated - not optimal, but better than nothing.

It's possible that Mozilla underestimates its share of tech-savvy[ish] users, which I imagine is becoming a greater proportion all the time.  The ability to use a custom sync server is one of just two program features that keeps me using Firefox, with the other being the extensive add-on support, which is a HUGE deal.  (Chrome for Android does not support add-ons at all, which in my case makes the web look and feel like a magazine - only worse, because things flash at you.)

More importantly than all that, I use Firefox because of Mozilla's commitment to the open web.  I would not like to see that commitment die with hard-coded and immutable references to specific servers, no matter who runs them.
(In reply to tw-bugs-firefox from comment #11)
> Thanks for the update, Nick...  Some grand words of support, there.  (I now
> realize that the remark about dev-fxacct probably was only directed at
> developers, so although I've successfully subscribed to it, I'll remain a
> mouse, there.)

This is sort-of, but not really, true: dev-fxacct is for discussion about the development of Firefox Accounts, but it's certainly not developers only.  It's a great place to influence the technical direction of Firefox Accounts.

> Even if Mozilla decides not to document or invest extra effort in custom
> sync server support or software, if at least a small amount could be
> directed toward an about:config option for configuring custom sync server
> parameters on Android, it'd be appreciated - not optimal, but better than
> nothing.
> 
> It's possible that Mozilla underestimates its share of tech-savvy[ish]
> users, which I imagine is becoming a greater proportion all the time.  The
> ability to use a custom sync server is one of just two program features that
> keeps me using Firefox, with the other being the extensive add-on support,
> which is a HUGE deal.  (Chrome for Android does not support add-ons at all,
> which in my case makes the web look and feel like a magazine - only worse,
> because things flash at you.)

It is possible that Mozilla does underestimate.  However, I think we're seeing the opposite: our user base is becoming less technically savvy.  That doesn't mean we should or will drop features attractive to our technically savvy users, but it does mean we have to fight for them :)

> More importantly than all that, I use Firefox because of Mozilla's
> commitment to the open web.  I would not like to see that commitment die
> with hard-coded and immutable references to specific servers, no matter who
> runs them.

Roger that.
Status update, the second.  After face-to-face discussions with some folks, valuable feedback from https://bugzilla.mozilla.org/show_bug.cgi?id=1003877#c11, and additional conversations about add-ons on dev-fxacct, I propose that on Fennec we:

1) expand the existing Accounts.jsm [1] launchSetup to include optional auth and token server URIs;

2) show (but do not allow editing) custom URIs in the native FxAccount*Activity UIs.

Part 1 will allow an add-on to start the setup process with the additional configuration; part 2 will meet our goal of least surprise: we never want a user to silently configure FxA + sync with malicious custom URLs.  This does not attempt to address any UI, nor does it attempt to help users test the status of their device-and-server combination.

[1] http://mxr.mozilla.org/mozilla-central/source/mobile/android/modules/Accounts.jsm

rnewman, ckarlof: what say you?
Flags: needinfo?(rnewman)
Flags: needinfo?(ckarlof)
(In reply to Nick Alexander :nalexander from comment #13)

> 1) expand the existing Accounts.jsm [1] launchSetup to include optional auth
> and token server URIs;

Let's put in a little effort to make that API somewhat future-proof, bearing in mind Chris's service proposal. I'd like to not have to distort the method signatures or whatever when we get a better way to do this.


> rnewman, ckarlof: what say you?

I'm fine with this approach.

My suggestion for how to do the add-on part (and I volunteer to build this if nobody else is motivated): this shouldn't be an add-on with UI, because that's a pain in the ass. (Checking values, localizing...)

The add-on should be a templated, restartless, self-uninstalling add-on, delivered by your Sync server/a web tool/your system administrator -- the place that already knows the URIs.

"Click here to set up Sync with this server".

(Let's face it: a user will just try installing the .xpi over and over if it fails, so we lose nothing by doing this.)

The add-on would simply immediately call Accounts.launchSetup with the right parameters, then uninstall itself. I think the kids these days are saying "boom".
Flags: needinfo?(rnewman)
I love this idea. The idea of setting up your own server, then visiting a known url on there to configure your device is compelling.
(In reply to Richard Newman [:rnewman] from comment #14)
> (In reply to Nick Alexander :nalexander from comment #13)
> 
> > 1) expand the existing Accounts.jsm [1] launchSetup to include optional auth
> > and token server URIs;
> 
> Let's put in a little effort to make that API somewhat future-proof, bearing
> in mind Chris's service proposal. I'd like to not have to distort the method
> signatures or whatever when we get a better way to do this.

rnewman and I discussed this briefly on #mobile, and we settled on something like:

launchSetup({"auth": ..., "services":{"sync":...}})

The idea is that in some brave future world, the services descriptor bundle will be fetched from the specified auth server.  Or perhaps we support auth + services.sync for the foreseeable future, but in the further future we add a discovery or metaserver entry point.

> > rnewman, ckarlof: what say you?
> 
> I'm fine with this approach.
> 
> My suggestion for how to do the add-on part (and I volunteer to build this
> if nobody else is motivated): this shouldn't be an add-on with UI, because
> that's a pain in the ass. (Checking values, localizing...)

I love this idea.  We'd want to expose something API-compatible with Accounts.jsm to desktop so that this could be uniform.
> The add-on should be a templated, restartless, self-uninstalling add-on, delivered by your Sync server/a web tool/your system administrator -- the place that already knows the URIs.

I like this idea for someone like ownCloud, who takes on the burden to make sure the add-on is configured correctly and hosted at the right place.

I'm skeptical it's a win for non-wizard enthusiasts unless the add-on is installed and configured as part of a one-click installer for FxA and Sync. As it stands now, it seems to me that the user must take on minor add-on development as an additional dependency.

A global add-on with simple UI seems like a good middle ground in the mean time, but I'd be happy if you proved me wrong.
Flags: needinfo?(ckarlof)
(In reply to Chris Karlof [:ckarlof] from comment #17)
> > The add-on should be a templated, restartless, self-uninstalling add-on, delivered by your Sync server/a web tool/your system administrator -- the place that already knows the URIs.
> 
> I like this idea for someone like ownCloud, who takes on the burden to make
> sure the add-on is configured correctly and hosted at the right place.

I would deliver the add-on XPI as part of fxa-auth-server.  Of course, if the user wants Mozilla-FxA + custom Sync, that's not so useful.
(In reply to Nick Alexander :nalexander from comment #18)

> I would deliver the add-on XPI as part of fxa-auth-server.  Of course, if
> the user wants Mozilla-FxA + custom Sync, that's not so useful.

May be better to have it as part of the tokenserver. That's where we were originally planning to put the discovery service, too - here's the things we offer tokens for.
From a testing point of view, I really like this.  I would make two requests for easier testing:

1. Please provide a URL or a parallel XPI/plug-in for "Help -- I've borked my settings.  Set me back to Mozilla defaults, please!" aka "I'm done testing now and would like to resume life as a civilian."

2. Make it very, very easy to tell which server(s) you're currently pointing to (I'm pretty sure this is covered by (2) in comment 13 above).
(In reply to Karl Thiessen [:kthiessen] from comment #20)
> From a testing point of view, I really like this.  I would make two requests
> for easier testing:

Thank you for this perspective.  It's important!

> 1. Please provide a URL or a parallel XPI/plug-in for "Help -- I've borked
> my settings.  Set me back to Mozilla defaults, please!" aka "I'm done
> testing now and would like to resume life as a civilian."

On Android, we're never going to support changing your settings mid-stream.  (We're not wired to invalidate caches, tokens, etc when you swap out various servers/services.  That's a lot of work for little gain.)  So this looks like "remove the Android Account" followed by creating a new one (but not using the special add-on).  On Desktop, it's a little funkier, 'cuz you can never really trust that prefs have remained constant.  I think disconnect followed by new setup is still the right model, but it's not my call.

> 2. Make it very, very easy to tell which server(s) you're currently pointing
> to (I'm pretty sure this is covered by (2) in comment 13 above).

I was thinking of during the setup flow; we'll definitely include the custom server URLs.  I wasn't really thinking of when the account had already been created; but for Android, this will require adding new custom server URL indicators in the status activity.  Thanks for the reminder!
Keywords: meta
Summary: Custom server support for Firefox Accounts on Android → [meta] Custom server support for Firefox Accounts on Android
Split because it would be nice to uplift Bug 1022748, and the other parts (Bug 1022749) need strings.
+1 for implement the option that you can use your own sync server again.
This has all landed.  Wow, a closed meta bug!
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
\o/
It doesn't happen often.
Congrats...
Product: Android Background Services → Firefox for Android
You need to log in before you can comment on or make changes to this bug.