968032 - Document how to run a sync1.5 server setup

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

We have a start on a host-your-own sync1.5 setup here:  https://github.com/mozilla-services/server-full2

Let's get it working properly, and add documentation to the howtos on docs.services.mozilla.com

Comment 1

[Richard Newman [:rnewman]]

TODO: file bugs for clients to support self-hosted servers!

Comment 2

[Ryan Kelly [:rfkelly]]

Attachment: doc-run-your-own-fxa.diff

Here's a start on this.  It's just a rough concept guide at this stage, pointing people to the appropriate fxa-*-server repos and their READMEs.  Definitely needs a lot of work, but we should get something simple up just to flag that yes, this is important to us.

Comment 3

[Ryan Kelly [:rfkelly]]

Attachment: syncserver.tar.gz

And here is a start on an all-in-one sync1.5 setup.  It's in the spirit of our current server-full repo, but I've tried to simplify things a bit since we're using a simper build process for our own deployments as well.

Basically it's a simple pyramid app that "mounts" tokenserver and syncstorage into its URL space, and fills in sensible defaults in the deployment settings to make them work well together.

It's not quite working yet, I need to figure out an appropriate default setup for the node-assignment backend.  But it's enough to give you an idea of what the end product will look like.

Comment 4

[Ryan Kelly [:rfkelly]]

> file bugs for clients to support self-hosted servers!

What do you have in mind here - simply exposing this as an option in the UI, or something more complicated?

Comment 5

[Richard Newman [:rnewman]]

(In reply to Ryan Kelly [:rfkelly] from comment #4)
> > file bugs for clients to support self-hosted servers!
> 
> What do you have in mind here - simply exposing this as an option in the UI,
> or something more complicated?

On Android this *has* to be an option in the UI -- about:config isn't relevant. On desktop, I'm ambivalent. I don't know exactly how we want to address this, but it seems that we need some combination of the following:

* I don't want my identity leaking to Mozilla. I want to run my own account server and all of the services I use.
* I don't mind having a Firefox account, but I want to use arbitrary third-party services with it, including self-hosting my own Sync server.

To do both of these requires service descriptor handling of some kind. We can manage just the former by using some kind of well-known URI to discover the Sync server from the account server.

uxwanted…

Comment 6

[Toby Elliott [:telliott]]

Comment on attachment 8372134 [details] [diff] [review]
doc-run-your-own-fxa.diff

Review of attachment 8372134 [details] [diff] [review]:
-----------------------------------------------------------------

Good start. There are a lot of configurations to be covered here, which is going to be a challenge.

::: source/fxa/index.rst
@@ +7,5 @@
> +The Firefox Accounts server provides a centralized database of all user
> +accounts for accessing Mozilla-hosted services.  It replaces the sync-specific
> +:ref:`registration <reg>` and :ref:`secure-registration <sreg>` services.
> +
> +Firefox Accounts support is included in Firefox version 29 and later.

I think one of the first things we should talk about is the choice you need to make between hosting your own FxA server, or using ours. It's going to come up all the time, and being able to express it up front as the simpler option versus the option for people who want to be totally disconnected will help with the messaging.

Right now, looking at this, I don't know which choice I should be making.

::: source/howtos/run-fxa.rst
@@ +17,5 @@
> +- nodejs: http://nodejs.org/download
> +
> +A self-hosted Firefox Accounts server requires two components: an auth-server
> +that manages the accounts database, and a content-server that hosts a web-based
> +user interface.

Is the content server required? Can I do direct db manipulation to avoid it? (similarly to how the auth server wasn't required if you were willing to issue some direct SQL)

@@ +52,5 @@
> +
> +You can ask for help:
> +
> +- in our Mailing List: https://mail.mozilla.org/listinfo/dev-fxacct
> +- on IRC (irc.mozilla.org) in the #fxa channel

as with sync server, I'd reverse these.

::: source/howtos/run-sync-1.5.rst
@@ +48,5 @@
> +
> +Get the latest version at https://github.com/mozilla-services/syncserver and
> +run the **build** command::
> +
> +    $ git clone https://github.com/mozilla-services/syncserver

Is this a server-full equivalent that has yet to be created?

I'm confused how the tokenserver fits in here.

@@ +109,5 @@
> +Once the server is launched, you need to tell Firefox about its location.
> +Go to "about:config", search for "services.sync.tokenServerURI" and change
> +its value to the URL of your server with a path of "token/1.0/sync/1.5":
> +
> +  - services.sync.tokenServerURI:  http://sync.example.com/token/1.0/sync/1.5

How does the tokenserver know where to point the user? Is it just "use the same host" automatically? (this seems OK for now, though we'll want full instructions eventually)

@@ +129,5 @@
> +Running behind a Web Server
> +===========================
> +
> +The built-in server should not be used in production, as it does not really
> +support a lot of load.

I've often wondered about this. We say they shouldn't use it, but for a single person, it's probably fine. Maybe we should flesh this out a little more. It would certainly cut down on a lot of the "I can't configure wsgi" questions we get!

@@ +145,5 @@
> +   For example, if your server will be located at http://example.com/ff-sync/,
> +   the public_url should be set to this value in your config file::
> +
> +       [syncserver]
> +       public_url = http://example.com/ff-sync/

Ah, so is this what the tokenserver is working off of?

It's a tough question figuring out how much of the tokenserver internals we want to expose to self-hosters.

@@ +187,5 @@
> +    CustomLog /var/log/apache2/example.com-access.log combined
> +    ErrorLog  /var/log/apache2/example.com-error.log
> +  </VirtualHost>
> +
> +We provide a **syncserver.wsgi** file for you convenience in the repository.

"your"

@@ +241,5 @@
> +==================
> +
> +* how to resrict new-user signups
> +* how to interoperate with a self-hosted accounts server
> +* periodic pruning of expired sync data

I don't think this one matters in self-hosted envs. Low priority.

@@ +250,5 @@
> +
> +Don't desitate to jump online and ask us for help:
> +
> +- in our Mailing List: https://mail.mozilla.org/listinfo/services-dev
> +- on IRC (irc.mozilla.org) in the #sync channel

I'd reverse these two. We'd rather have people try IRC first.

Comment 7

[James Bonacci [:jbonacci]]

Just adding a note for the QA team, based on today's discussion with Ryan.
All of this documentation will have to include some details on either
1. using Mozilla's (Production) FxA auth and content service
2. setting up a user-defined auth and content service in addition to the Sync/TS backend

Comment 8

[Ryan Kelly [:rfkelly]]

> Is the content server required? Can I do direct db manipulation to avoid it? (similarly to how the auth server wasn't required if you were willing to issue some direct SQL)

It hosts the "sign in to firefox" UI and I'm pretty sure it uses a baked-in fxa-auth-server URL to do so.  There may be a way to hack around it but I think it's probably a good idea for a smooth flow.

Comment 9

[Richard Newman [:rnewman]]

(In reply to Ryan Kelly [:rfkelly] from comment #8)

> It hosts the "sign in to firefox" UI and I'm pretty sure it uses a baked-in
> fxa-auth-server URL to do so.  There may be a way to hack around it but I
> think it's probably a good idea for a smooth flow.

identity.fxaccounts.remote.uri is the pref for the remote page, fwiw.

Comment 10

[Ryan Kelly [:rfkelly]]

> > +The built-in server should not be used in production, as it does not really
> > +support a lot of load.
> I've often wondered about this. We say they shouldn't use it, but for a single person, it's probably
> fine. Maybe we should flesh this out a little more. It would certainly cut down on a lot of the "I
> can't configure wsgi" questions we get!

Even better, we should just install gunicorn as part of the build and run it using that.  It's just and easy and will happily serve production traffic.

Comment 11

[Ryan Kelly [:rfkelly]]

Committed with suggested edits here: https://github.com/mozilla-services/docs/commit/c4b54242c37ddde980690906a01391333cde5ad9

I'll leave this bug open while we iterate on the details.

Comment 12

[Loic]

Do you know if the new sync1.5 server API will be available for Firefox 29? Because, I guess some people will be disappointed if they learn the new Sync cannot be used with their own server.

Comment 13

[Toby Elliott [:telliott]]

https://docs.services.mozilla.com/howtos/run-sync-1.5.html has the gory details.

Comment 14

[Pin Zhang [:pzhang] (inactive)]

Any doc for how to setup node-assignment backend?

Comment 15

[Ryan Kelly [:rfkelly]]

> Any doc for how to setup node-assignment backend?

Minimal, since self-hosting setups almost certainly won't need it.  We should at least add some pointers to show people what the pieces are.

The tokenserver repo has *some* documentation:

  https://github.com/mozilla-services/tokenserver/tree/master/docs/source

And the node-assignment backend code itself may be useful:

  https://github.com/mozilla-services/tokenserver/tree/master/tokenserver/assignment/sqlnode

Comment 16

[Ryan Kelly [:rfkelly]]

Folks are successfully self-hosting this in the wild, calling this bug done