579247 - Firefox Home should allow configuration for an non-Mozilla sync server

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Firefox Home, defect)

Description

[Les Orchard [:lorchard]]

Both desktop and mobile Firefox Sync accept the URL for a non-Mozilla sync server. I'd like to use my own sync server with Firefox Home, but there doesn't appear to be a way to configure the server URL.

FWIW, I wrote a sync server for Google App Engine:

http://decafbad.com/blog/2010/07/05/firefox-sync-server-on-google-app-engine

Since freedom from lock-in to Mozilla servers seems to be an important part of the Sync story, this seems like an important (though admittedly obscure) feature to have eventually.

Comment 2

[Les Orchard [:lorchard]]

Attachment: First stab at a patch to add a settings bundle for a custom sync server

So, I took a peek at the Firefox Home source and came up with this patch to allow custom server config in the Settings app:

http://hg.mozilla.org/users/lorchard_mozilla.com/fx-home-patches/file/eed624281079/bug-579247-custom-server

My Cocoa/ObjC is rusty, so feel free to beat up the code and its author.

Also, I know Apple's docs say you shouldn't have both in-app and Settings app settings, but I think it's not awful in this case.  Users with no need for an alternate sync server can follow the out-of-box setup flow, while more advanced users can dig around for the advanced options.

It seems to work fine against my custom server in the simulator. I don't have a dev cert, so I can't install on my iPod Touch to try for real.

The one gotcha I ran into is that my server seems not to offer a sortindex in WBO's when the value is zero. That's a server bug for sure - but it crashed Firefox Home, which seems undesirable. So, part of this patch looks for that, though it may not do it in the best way.

Comment 3

[Zandr Milewski [:zandr]]

Now that Firefox Home is released, this blocks any future server releases. If we can't test the iPhone on stage, we can't push anything to production.

Custom builds aren't really a solution, as they are time-consuming to build, time-consuming to install, error-prone, and hard to identify.

The default server really should be considered source code, so if you change that and spin a build, you aren't really testing the released product.

It should be said that the same is true of Fennec... does Fennec have about:config?

Comment 4

[Philipp von Weitershausen [:philikon]]

(In reply to comment #3)
> It should be said that the same is true of Fennec... does Fennec have
> about:config?

It does, at least on my OSX build.

Comment 5

[William Reynolds [:williamr]]

(In reply to comment #4)
> It does, at least on my OSX build.

about:config confirmed on my Android build as well.

Comment 6

[Dan Walkowski]

(In reply to comment #2)
> Created attachment 457857 [details] [diff] [review]
> First stab at a patch to add a settings bundle for a custom sync server
> 
> So, I took a peek at the Firefox Home source and came up with this patch to
> allow custom server config in the Settings app:
> 
> http://hg.mozilla.org/users/lorchard_mozilla.com/fx-home-patches/file/eed624281079/bug-579247-custom-server
> 
> My Cocoa/ObjC is rusty, so feel free to beat up the code and its author.
> 
> Also, I know Apple's docs say you shouldn't have both in-app and Settings app
> settings, but I think it's not awful in this case.  Users with no need for an
> alternate sync server can follow the out-of-box setup flow, while more advanced
> users can dig around for the advanced options.
> 
> It seems to work fine against my custom server in the simulator. I don't have a
> dev cert, so I can't install on my iPod Touch to try for real.
> 
> The one gotcha I ran into is that my server seems not to offer a sortindex in
> WBO's when the value is zero. That's a server bug for sure - but it crashed
> Firefox Home, which seems undesirable. So, part of this patch looks for that,
> though it may not do it in the best way.


I've added the check for nil sortindex in FF Home 1.0.1, changeset 1464c1b8098a

Comment 7

[Tony Chung [:tchung]]

fixed per comment 6

Comment 8

[Tony Chung [:tchung]]

Verified fix on checkin

Comment 9

[Dan Walkowski]

custom servers are NOT fixed for 1.0.1.

they are mostly supported at the top of the tree, well enough for testing, but there is still work to be done.

Comment 10

[Tony Chung [:tchung]]

(In reply to comment #6)
> 
> I've added the check for nil sortindex in FF Home 1.0.1, changeset 1464c1b8098a

Changeset linky: http://hg.mozilla.org/services/fx-home/rev/1464c1b8098a

Comment 11

[Les Orchard [:lorchard]]

One potential problem:

http://hg.mozilla.org/services/fx-home/rev/c8e68b10f333#l4.14

I noticed the "Node Query Suffix" is "user/1/%s/node/weave", but it should be
"user/1.0/%s/node/weave".  

The support for "1" rather than "1.0" works as a fix for an old client bug, if
I recall the conversations.  Using "1.0" is the intended Sync API version - at
least as far as the docs seem concerned:

https://wiki.mozilla.org/Labs/Weave/Sync/1.0/Setup#Setting_up_the_Server

Additionally, I think the Weave Minimal Server will throw a "Function not
found" error for any version not 0.5 or 1.0.

http://tobyelliott.wordpress.com/2009/09/11/weave-minimal-server/

Comment 12

[Dan Walkowski]

Thanks, I'll make that change to 1.0, but otherwise, does it work with missing sort indices?

Comment 13

[Les Orchard [:lorchard]]

(In reply to comment #12)
> Thanks, I'll make that change to 1.0, but otherwise, does it work with missing
> sort indices?

Looks like it works fine for me with that change!  I just updated, tweaked 1 -> 1.0, built it, logged in, and sync'd 2000 history items from my App Engine sync server.

Comment 14

[Tony Chung [:tchung]]

Moving FFHome-related bugs to new component -> Firefox Home

Comment 15

[Ragavan S [:rags]]

Should be fixed in 1.0.2.

Comment 16

[Dan Walkowski]

Is fixed in 1.0.2, changset: c8e68b10f333

Comment 17

[Tony Chung [:tchung]]

Dan, can you make sure you are adding entire path to the changeset next time?  This is standard nomenclature for bugzilla comments.

for this case, http://hg.mozilla.org/services/fx-home/rev/c8e68b10f333

Comment 18

[Tony Chung [:tchung]]

Verified fix on FF Home 1.0.2b3

Comment 19

[Jens Maus]

Sorry guys, but I have to say that I am quite **** because of FF Home 1.0.2 not being released at the App Store yet. I am still waiting to get custom server functionality in FF Home because otherwise the app is useless for me. In addition, I can only hope once FF Home 1.0.2 is released in the App Store that servers with own SSL certificates are supported as well or otherwise this change won't help really for most of the users.

Sorry if this sounds a bit harsh, but I can not understand that the issue seems to be fix more than a week now and still no updated being distributed to the App Store.

Comment 20

[Ragavan S [:rags]]

(In reply to comment #19) 

> Sorry if this sounds a bit harsh, but I can not understand that the issue seems
> to be fix more than a week now and still no updated being distributed to the
> App Store.

Jens - the 1.0.2 release is a pretty major release. Besides various other fixes, it is also our first localized release. We are currently waiting on the localization teams to give it another look to make sure the strings look OK in their respective languages. Once we get that, we should be ready to submit 1.0.2 to the App Store. I expect that we will be able to submit to the App Store sometime next week. 

In the meantime, if you have access to a Mac, feel free to grab the top of the tree from http://hg.mozilla.org/services/fx-home/ and use the iPhone simulator included in the iPhone SDK (XCode) to test it. 

Sorry that you are frustrated, but we are moving as fast as we can with what's a pretty new development/distribution process for us.

Comment 21

[Net Nut]

I got 1.0.2 from the app store today, and it does not appear to have a place to put in a custom server in the settings. Wasn't it supposed to be in 1.0.2 ?

Thanks for the info.

Comment 22

[Dan Walkowski]

It's in a custom Settings pane.  Go to the Settings app, then scroll down to the bottom, you will see a Firefox Home pane.

Comment 23

[Jens Maus]

Here I tried the whole day to get the new 1.0.2 release (thanks for having it out now, btw) working with my Weave minimal server install. However, I cannot get it working. I can see the following entries in my apache server log when I try to connect via firefox home to my minimal server:

X.X.X.X - USER [19/Sep/2010:16:17:37 +0200] "GET /weave/user/1.0/USER/node/weave HTTP/1.1" 404 20 "-" "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"

After that I get an error on my iphone telling me that the authentication failed.

However, if I use Firefox Sync in Firefox directly I can perfectly enter the same URL and credentials and see the authentication process continuing:

X.X.X.X - USER [19/Sep/2010:16:20:34 +0200] "GET /weave/user/1.0/USER/node/weave HTTP/1.1" 404 20 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; de; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
X.X.X.X - USER [19/Sep/2010:16:20:34 +0200] "GET /weave/1.0/USER/info/collections HTTP/1.1" 200 230 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; de; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"

So it seems the firefox sync plugin checks not only the /user path but also tries to access the collection directly while firefox home directly cancels the connection request because the /user requests is answered with 404

Would be nice to see Firefox Home working with a minimal server installation as well. Any advice?

Comment 24

[Philipp von Weitershausen [:philikon]]

(In reply to comment #23)
> So it seems the firefox sync plugin checks not only the /user path but also
> tries to access the collection directly while firefox home directly cancels the
> connection request because the /user requests is answered with 404

Right. A 404 from the /user/... API indicates to the client that the server doesn't support the user API and that it should proceed to directly use the sync API. This is how Firefox Sync does it and what Firefox Home should do as well. I'm guessing it doesn't :/ Please file a new bug for this.

One way to solve this for now is to extend your minimal server installation to support at least the /node/weave part of the user API. IOW have your server return its own URL for all /user/1.0/<USER>/node/weave calls. Alternatively you can use the new Python implementation at http://bitbucket.org/tarek/sync-server which implements the user API as well.

Comment 25

[Jens Maus]

(In reply to comment #24)
> (In reply to comment #23)
> > So it seems the firefox sync plugin checks not only the /user path but also
> > tries to access the collection directly while firefox home directly cancels the
> > connection request because the /user requests is answered with 404
> 
> Right. A 404 from the /user/... API indicates to the client that the server
> doesn't support the user API and that it should proceed to directly use the
> sync API. This is how Firefox Sync does it and what Firefox Home should do as
> well. I'm guessing it doesn't :/ Please file a new bug for this.

Very frustrating that the 1.0.2 release has only been tested with a full fledged server implementation and not with the very commonly used minimal server implementation. And I have my additional doubts that it works with https connections to servers running self-signed certificates because I cannot even get it working with my other server installation running the user API. However, I will try to open a new bug report for that item in the hope it gets fixed for 1.0.3 - however, that means another month waiting for the full release cycle through app store & co :/

> One way to solve this for now is to extend your minimal server installation to
> support at least the /node/weave part of the user API. IOW have your server
> return its own URL for all /user/1.0/<USER>/node/weave calls. Alternatively you
> can use the new Python implementation at http://bitbucket.org/tarek/sync-server
> which implements the user API as well.

Thanks for the information, however a documentation for that yet another firefox sync server implementation is missing. So if you have any docs, howto guide it would be appreciated.

Comment 26

[pkytiuqdwwq]

> however, that means another month waiting for the full release cycle
> through app store & co :/

if you are using this (http://tobyelliott.wordpress.com/2009/09/11/weave-minimal-server/) Weave Minimal Server use this workaround to use the iPhone App:

Find Line 122 in index.php: 
list($version, $username, $function, $collection, $id) = explode('/', $path.'//');

After this line add:
if($version == 'user')
{	
	header("Content-type: text/plain");
	echo 'https://YOURDOMAIN/weave/';
	exit;
}

Comment 27

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

I filed a bug for this: https://bugzilla.mozilla.org/show_bug.cgi?id=597865

Comment 28

[Jens Maus]

(In reply to comment #26)
> > however, that means another month waiting for the full release cycle
> > through app store & co :/
> 
> if you are using this
> (http://tobyelliott.wordpress.com/2009/09/11/weave-minimal-server/) Weave
> Minimal Server use this workaround to use the iPhone App:
> 
> Find Line 122 in index.php: 
> list($version, $username, $function, $collection, $id) = explode('/',
> $path.'//');
> 
> After this line add:
> if($version == 'user')
> {    
>     header("Content-type: text/plain");
>     echo 'https://YOURDOMAIN/weave/';
>     exit;
> }

With this work around it still doesn't work. However, now it seems to pass the /user/... API GET, but fails right away after that:

X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET /weave/user/1.0/USERNAME/node/weave HTTP/1.1" 200 34 "-" "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"
X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET /weave/1.0/USERNAME/storage/meta/global HTTP/1.1" 200 490 "-" "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"

So I still get an "authentication failed" response from the iPhone App, that's all.

Comment 29

[Philipp von Weitershausen [:philikon]]

(In reply to comment #28)
> With this work around it still doesn't work. However, now it seems to pass the
> /user/... API GET, but fails right away after that:
> 
> X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> /weave/user/1.0/USERNAME/node/weave HTTP/1.1" 200 34 "-" "Firefox%20Home/1.0.2
> CFNetwork/485.10.2 Darwin/10.3.1"
> X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> /weave/1.0/USERNAME/storage/meta/global HTTP/1.1" 200 490 "-"
> "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"

I don't see a failure here. Both requests are answered with a 200 response code.

Comment 30

[Jens Maus]

(In reply to comment #29)
> (In reply to comment #28)
> > With this work around it still doesn't work. However, now it seems to pass the
> > /user/... API GET, but fails right away after that:
> > 
> > X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> > /weave/user/1.0/USERNAME/node/weave HTTP/1.1" 200 34 "-" "Firefox%20Home/1.0.2
> > CFNetwork/485.10.2 Darwin/10.3.1"
> > X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> > /weave/1.0/USERNAME/storage/meta/global HTTP/1.1" 200 490 "-"
> > "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"
> 
> I don't see a failure here. Both requests are answered with a 200 response
> code.

I know, still it returns an authentication failure to me while the sync plugin for firefox works perfectly with the same server installation.

Comment 31

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

(In reply to comment #30)
> (In reply to comment #29)
> > (In reply to comment #28)
> > > With this work around it still doesn't work. However, now it seems to pass the
> > > /user/... API GET, but fails right away after that:
> > > 
> > > X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> > > /weave/user/1.0/USERNAME/node/weave HTTP/1.1" 200 34 "-" "Firefox%20Home/1.0.2
> > > CFNetwork/485.10.2 Darwin/10.3.1"
> > > X.X.X.X - USERNAME [19/Sep/2010:18:10:36 +0200] "GET
> > > /weave/1.0/USERNAME/storage/meta/global HTTP/1.1" 200 490 "-"
> > > "Firefox%20Home/1.0.2 CFNetwork/485.10.2 Darwin/10.3.1"
> > 
> > I don't see a failure here. Both requests are answered with a 200 response
> > code.
> 
> I know, still it returns an authentication failure to me while the sync plugin
> for firefox works perfectly with the same server installation.

There is a bug in 1.0.2 that makes custom servers that are *not* on httpS always fail. This is fixed in 1.0.3.

There is no reported bug for it: I ran into it when testing 1.0.3 and simply fixed it.

Comment 32

[Jens Maus]

(In reply to comment #31)
> (In reply to comment #30)
> > (In reply to comment #29)
> > I know, still it returns an authentication failure to me while the sync plugin
> > for firefox works perfectly with the same server installation.
> 
> There is a bug in 1.0.2 that makes custom servers that are *not* on httpS
> always fail. This is fixed in 1.0.3.
> 
> There is no reported bug for it: I ran into it when testing 1.0.3 and simply
> fixed it.

Ok. thanks for the fix. And have you checked also that httpS works with self-signed certificates as well so that we don't need another 1.0.4 to have all that own server setups working?

Comment 33

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

The current code does not support self-signed certs and will likely fail on them.

Unfortunately to correctly implement this, we will need some bigger refactorings in the HTTP code. I doubt I will be able to squeeze that in for the 1.0.3 release.

I will discuss it with those in charge.

Comment 34

[John Affleck]

I'm not sure how well advertised this is, so I'll add it here as a workaround for the self-signed issue. You can import certificates into an iOS device via either email or (I believe) safari in such a way that they are accepted as trusted. Doing this allowed me to get firefox home up and running on my own sync server over SSL with self-signed certificates. The steps I followed were:
- Converted both the site certificate and the CA certificates to the iOS friendly DER format:
  openssl x509 -in ca.crt -inform PEM -out ca.der -outform der
  openssl x509 -in site.crt -inform PEM -out site.der -outform der
- Mailed both certificates to my iPhone, opened the attachments and installed the certificates
- Pointed firefox home at https://weave.<mydomain>.net

..and everything worked. I'm not entirely positive that you need both the CA and the site certificate installed.

I got the original instructions from:
http://www.experts-exchange.com/Apple/Hardware/iPhone/A_2402-import-self-signed-certificates-into-iPhone.html
..amongst other places.

Comment 35

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Nice howto John. I'll ask if we can include this on the support site.

In a future version of Firefox Home we will have better support for self-signed certificates. Until then this looks like a good workaround.

Comment 36

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Getting reports that this is not working

http://support.mozilla.com/en-US/questions/758124

Comment 37

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Er, is the WONTFIX an accident?

Comment 38

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Yes sorry. This should be FIXED. I did file a separate bug for self-signed certificates.