1539282 - "send tab to device" doesn't use custom sync server

Status: NEW

(Firefox :: Sync, defect, P5)

Description

[funni.clonk]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

I started my FF and entered my masterpassword. While another program reported that the certificate of my server (with TLS webproxy for custom ff sync server) was expired today a tab (send to that device) was opened.
Unfortunally I don't have the ressources to check that at the moment. My access.log (webserver) has entries in the sync section with the device I sent the tab from (FF65).

Actual results:

FF seems to not validate the TLS connection with custom sync servers.

Expected results:

FF should validate these connections.

Comment 1

[:Gijs (he/him)]

Is the cert for the server one that was created by a publicly trusted CA? Or are you self-hosting and is the cert one you self-signed? (Can you attach the public portions of the certificate in question? )

Comment 2

[funni.clonk]

It's a nginx-webserver with a letsencrypt certificate so it's a trusted CA. The certificate expired yesterday. The first messages I got were at about 15 o clock. Sending and receiving tab happened later.
Attachment following.

Comment 3

[funni.clonk]

Attachment: Letsencrypt CA X3

Here the CA

Comment 4

[Daniel Veditz [:dveditz]]

(In reply to funni.clonk from comment #0)

While another program reported that the certificate of my server (with TLS webproxy for custom ff sync server) was expired today a tab (send to that device) was opened.

Did you open that server in the FIrefox that was doing the Sync? If you granted an exception to the expired certificate so you could see the site you are telling Firefox "I trust this certificate", and it would then also be trusted for Sync.

Comment 5

[funni.clonk]

Nope I didn't, on both PCs I just browsed and after a while I was asked in both cases for the masterpassword (because FF wanted to sync). I also sync a calendar with TB CalDAV. TB gave me the well-known certificate warning on the sender-pc. Caused by this warning I aborted the calendar's synching process and thought regardless of Thunderbird FF would not sync with that expired certificate.
So i got the warning, but it was in TB, it was caused by the CalDAV sync and i didn't grant the certificate (neither in FF nor in TB).
Maybe FF made a mistake while calculating the certificate's expire-timestamp so it was valid for the Firefox.

Comment 6

[:Gijs (he/him)]

(In reply to funni.clonk from comment #3)

Created attachment 9053881 [details]
Letsencrypt CA X3

Here the CA

This is the CA cert for letsencrypt, not the public part of the now-expired cert you were using for your own server...

Comment 7

[:Gijs (he/him)]

Was the certificate already expired at the point where Firefox started, or did it expire while Firefox was running?

Comment 8

[:Gijs (he/him)]

Dana, is there any other data that would help here? I'm pretty sure sync uses our regular https primitives (ie necko) and will validate certs accordingly...

Reporter, are you still able to reproduce, ie is Firefox still connecting to the server with the still-expired cert?

Comment 9

[funni.clonk]

Attachment: public key screenshot.png

(In reply to :Gijs (he/him) from comment #7)

Was the certificate already expired at the point where Firefox started, or did it expire while Firefox was running?
I didn't uploaded the x.509 because of privacy reasons. Here I uploaded a screenshot of the expiration date.(In reply to :Gijs (he/him) from comment #7)

Was the certificate already expired at the point where Firefox started, or did it expire while Firefox was running?
The sender was started at 17:49, the first successful internet connection was at 17:51 (half an hour before the certificate expired). I probably started FF right after that and entered the master password at the sender before the certificate expired (it expired while sender was running). The sender was powered off at 19:28.

I guarantee that the receiving computer was only started after 21 o'clock (from ACPI status S5), cause I drove home for an hour after the transmitter turned off. The tab was received after entering the masterpassword without any warning.

Comment 10

[:Gijs (he/him)]

(In reply to :Gijs (he/him) from comment #8)

Reporter, are you still able to reproduce, ie is Firefox still connecting to the server with the still-expired cert?

Still curious about this.

Comment 11

[funni.clonk]

Sorry for the wrong formatting from from comment #9, now I use the preview:
(In reply to :Gijs (he/him) from comment #6)

(In reply to funni.clonk from comment #3)

Created attachment 9053881 [details]
Letsencrypt CA X3

Here the CA

This is the CA cert for letsencrypt, not the public part of the now-expired cert you were using for your own server...

I didn't uploaded the x.509 because of privacy reasons. See above for a screenshot of the certificate expiration date.

(In reply to :Gijs (he/him) from comment #7)

Was the certificate already expired at the point where Firefox started, or did it expire while Firefox was running?

The sender was started at 17:49, the first successful internet connection was at 17:51 (half an hour before the certificate expired). I probably started FF right after that and entered the master password at the sender before the certificate expired (it expired while sender was running). The sender was powered off at 19:28.

I guarantee that the receiving computer was only started after 21 o'clock (from ACPI status S5), cause I drove home for an hour after the transmitter turned off. The tab was received after entering the masterpassword without any warning.

Comment 12

[funni.clonk]

(In reply to :Gijs (he/him) from comment #10)

(In reply to :Gijs (he/him) from comment #8)

Reporter, are you still able to reproduce, ie is Firefox still connecting to the server with the still-expired cert?

Still curious about this.

I updated the certificate already but I'll try to reproduce that, I have the expired cert. It's not a 1:1 reproduce because I'll send with a valid certificate and switch then to an old certificate while the reveicer (started after) already knows the new certificate.

Comment 13

[funni.clonk]

Okay I could reproduce it:
0. Both devices were online and connected to the sync service

1. Closed FF on receiver (no logoff, maintenance service not running)
2. Sent the tab (let sender online)
3. Switched to the old certificate
4. Receiver's and sender's services (depending from the webserver) warned me that the certificate was expired
5. Opened Firefox
6. -> Tab was shown immediately (for whatever reason without entering masterpassword)
7. -> (about 10 seconds later asked for the masterpass)

In any case it must not work because I've HSTS enabled. I guess there are maybe sessionkeys between the sync server and FF (so that the masterpassword wasn't requested).
If I request the website manually I get a warning and I get the message that HSTS is enabled so that I cannot grant something.

Comment 14

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to :Gijs (he/him) from comment #8)

Dana, is there any other data that would help here? I'm pretty sure sync uses our regular https primitives (ie necko) and will validate certs accordingly...

Reporter, are you still able to reproduce, ie is Firefox still connecting to the server with the still-expired cert?

A packet trace would help. My understanding is on android, sync actually doesn't use our network stack because it's implemented as an android service or something. I gather this is on desktop, though?

Comment 15

[funni.clonk]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #14)

(In reply to :Gijs (he/him) from comment #8)
A packet trace would help. My understanding is on android, sync actually doesn't use our network stack because it's implemented as an android service or something. I gather this is on desktop, though?

Both computers are on Windows 10 1809 with FF 65 (sender) and 66 (reveceiver)
I can do a packet trace but that will take some time because I am not at home now.

Comment 16

[Daniel Veditz [:dveditz]]

Certificate details are checked during the handshake (when the connection is made), not during a session (e.g. if the cert expired while you're sending stuff it'll keep sending to that host. Not sure if the cert is re-checked if we do session resumption (though I'd think so?).

Comment 17

[funni.clonk]

(In reply to Daniel Veditz [:dveditz] from comment #16)

Certificate details are checked during the handshake (when the connection is made), not during a session (e.g. if the cert expired while you're sending stuff it'll keep sending to that host. Not sure if the cert is re-checked if we do session resumption (though I'd think so?).

Yeah that's right, a connected TLS session will not verify the x.509 again when the symetric TLS session keys are negotiated (Handshake). But please note that (when I discovered the issue) the recveiver was turned off over about one day and was turned on after I sent the FF-tab. So there was a new TCP connection (when starting FF) and caused by this a new TLS session (means new x.509-validation, handshake ectera pp.).
What do you mean with "session resumption"?

Comment 18

[funni.clonk]

I have a dark suspicion: There is no connection to the sync server in the Wireshark dump. With invalid certificates I cannot sync the opened tabs between computers (empty remote-tabs-list) but sending tabs does still works. Everywhere is told that all sync features (exempt account) are served by the custom sync server.

Is it possible that tabs that are sent via "Send tab to device" do not run across the sync server from identity.sync.tokenserver.uri? if that's how it is: Why?

Comment 19

[:Gijs (he/him)]

(In reply to funni.clonk from comment #18)

I have a dark suspicion: There is no connection to the sync server in the Wireshark dump. With invalid certificates I cannot sync the opened tabs between computers (empty remote-tabs-list) but sending tabs does still works. Everywhere is told that all sync features (exempt account) are served by the custom sync server.

Is it possible that tabs that are sent via "Send tab to device" do not run across the sync server from identity.sync.tokenserver.uri? if that's how it is: Why?

Mark/Ryan, can you answer this? I don't know enough about sync's internals.

Comment 20

[Ryan Kelly [:rfkelly]]

Indeed, that's correct; the new send-tab system uses features of the Firefox Account service to send tabs to connected devices in a more timely manner, rather than the old system that wrote them into the sync storage service.

I can see how this would be surprising behaviour in the context of a self-hosted sync server using the mozilla-hosted account server, and depending on the reasons for self-hosting, a violation of expected trust boundaries. I'll have to think about what the right path forward is here...

Comment 21

[funni.clonk]

(In reply to Ryan Kelly [:rfkelly] from comment #20)

Indeed, that's correct; the new send-tab system uses features of the Firefox Account service to send tabs to connected devices in a more timely manner, rather than the old system that wrote them into the sync storage service.

I can see how this would be surprising behaviour in the context of a self-hosted sync server using the mozilla-hosted account server, and depending on the reasons for self-hosting, a violation of expected trust boundaries. I'll have to think about what the right path forward is here...

In the actual case, please do this, or give the user the possibility to change it, e.g. via a second about:config entry. This was completely unknown to me.

Comment 22

[funni.clonk]

(In reply to funni.clonk from comment #21)

This was completely unknown to me.

I thought it meets the sync-server because I'have the devices in the list that I have just in e.g. "Tabs von anderen Geräten" ("Tabs from other devices") and those are synced via the self hosted sync-server.

Comment 23

[Ryan Kelly [:rfkelly]]

In the actual case, please do this, or give the user the possibility to change it, e.g. via a second about:config entry.

One short-term option would be to toggle the identity.fxaccounts.commands.enabled pref to false, which will cause Firefox to stop using the new send-tab implementation in favour of the old one.

We could consider making this false automatically if the user has configured a self-hosted sync server but is using the default account server. Mark, thoughts?

Comment 24

[Mark Hammond [:markh] [:mhammond]]

(In reply to Ryan Kelly [:rfkelly] from comment #23)

We could consider making this false automatically if the user has configured a self-hosted sync server but is using the default account server. Mark, thoughts?

We could do this, although there are likely to be implications down the road. These commands are intended to replace sending tabs via the old "client collections" technique, and while we currently support both old and new, our expectation was that the "old" would go away at some point (eg, it's not clear Fenix will support commands at all). This means we will probably be forced into one of:

• When we declare the "old" is dead, users with this pref set to false should probably have the entire send-tab UI functionality hidden, or otherwise do something to make it obvious that it doesn't work.

• We commit to supporting the "old" forever, including in new products.

Given this is such a tiny fraction of our users, the "otherwise do something to make it obvious that it doesn't work" might be able to be ugly - eg, an alert or similar.

Comment 25

[Ryan Kelly [:rfkelly]]

We commit to supporting the "old" forever, including in new products.

Let's not do this one...

Comment 26

[funni.clonk]

(In reply to Mark Hammond [:markh] from comment #24)

• We commit to supporting the "old" forever, including in new products.

Yes, please.

(In reply to Ryan Kelly [:rfkelly] from comment #23)

In the actual case, please do this, or give the user the possibility to change it, e.g. via a second about:config entry.

One short-term option would be to toggle the identity.fxaccounts.commands.enabled pref to false, which will cause Firefox to stop using the new send-tab implementation in favour of the old one.

I didn't find "identity.fxaccounts.commands.enabled" on FF for Android. Does FF Android recognize the value, if I add it manually? Is it implemented in FF Android?

Comment 27

[Ryan Kelly [:rfkelly]]

I didn't find "identity.fxaccounts.commands.enabled" on FF for Android.
Does FF Android recognize the value, if I add it manually? Is it implemented in FF Android?

This value does not exist on Firefox for Android, it will always use the "old" send-tab mechanism because we haven't implemented the new one there.

We commit to supporting the "old" forever, including in new products.
Yes, please.

I don't think that's going to be sustainable; we should instead try to figure how to make the new system meet your needs.

Can you say more about why you're self-hosting sync but not self-hosting the account service, and what properties of your current setup are valuable to you? From the above comments I interpret it as being about privacy/control over the data, but I could easily be wrong.

Comment 28

[funni.clonk]

Attachment: Workflow sync-setup improvement

(In reply to Ryan Kelly [:rfkelly] from comment #27)

Can you say more about why you're self-hosting sync but not self-hosting the account service, and what properties of your current setup are valuable to you? From the above comments I interpret it as being about privacy/control over the data, but I could easily be wrong.

Thank you for the question :)
I thought very critical about sync, when you released it a few years ago. I had (and have) strong privacy concerns. I don't want to say that you would sell all my data but I don't know under which law the data are protected from government's enforcement (See e.g. U.S. Prism project). Then I discovered the sync server based on self hosting. With an existing infrastructure (Webserver was already running) I set up the sync server on a server in my house. Now I have full control over the synced data and can see, which clients accessed to it, without risk of log-manipulation by law enforcement (if the mozilla accounts server gives access). Another point is that I am independent of mozilla services (maintenance, DDOS etc.) with a self hosted server. With my own sync server I am sure the risk of data being stolen is less than with mozilla services: With this centralized model of mozilla services an attacker has just "one" target, he knows the target because it's public. While you serve for many users you'll be attacked more then one time a day I guess. Of course this argument doesn't protect against a targeted attack against my self hosted server. No system is safe.

I also thought about setting up an accounts-server but I was too lazy for that because of the following reasons:
*I thought accounts.firefox doesn't know, whats the sync-server's address. So if the police officer knocks at your door you can shrug your shoulders.
*Also if you had my sync server address, I would recognize access trough the logs (I am using a script that warns me if "unnormal" ip ranges access to the sync server).
*Third point was, that I currently have to set one about:config entry (tokenserver.uri). I don't want to set all account-entries also. Then I would need to have a large noteblock to carry with me :).
*The whole data are synced via my server anyway (that was before I read comment #20)

Generally, based on my experiences with setting up sync, I have a proposal, a completely different concept for setting up sync (see attachment). With this workflow the user (me and the other nerds) doesn't need to manipulate the about:config, he just needs to know the domain of his self hosted server. At the moment mozilla makes it hard for self hosting users (See point three).
How do you feel about make it easier for users to use a self hosted server when setting up FF client?

Comment 29

[Ryan Kelly [:rfkelly]]

I am generally in favour of it, although it's hard to prioritize relative to other engineering work. There's actually already the start of an "autodiscover" mechanism available if you're running your own account service, as described here:

https://mozilla-services.readthedocs.io/en/latest/howtos/run-fxa.html

With the "identity.fxaccounts.autoconfig.uri" pref.

(leaving ni? myself because I want to come back to this)

Comment 30

[Daniel Veditz [:dveditz]]

Given that this is functioning "as designed" and that the real issue is unexpected implications of that design leading to an unwelcome surprise in custom deployments, can this be made a public bug? It doesn't appear to fit the mold of a "Firefox security bug" where hiding it is protecting our users. It might actually be protective of other folks like funni.clonk if they knew about this and could take it into account.

Comment 31

[Ryan Kelly [:rfkelly]]

can this be made a public bug?

Yes, please do.

Comment 32

[funni.clonk]

(In reply to Ryan Kelly [:rfkelly] from comment #29)

I am generally in favour of it, although it's hard to prioritize relative to other engineering work. There's actually already the start of an "autodiscover" mechanism available if you're running your own account service, as described here:

Yeah of course I understand, but please store it for medium-term milestones :)

https://mozilla-services.readthedocs.io/en/latest/howtos/run-fxa.html

With the "identity.fxaccounts.autoconfig.uri" pref.

(leaving ni? myself because I want to come back to this)

Sorry I didn't understand your last sentence.

So if I had account+sync server set up (account server configured for "autoconfig" (probably with sync-server-url)) I just need to set identity.fxaccounts.autoconfig.uri to the accounts server that would automatically assign the sync server?

Comment 33

[:Gijs (he/him)]

Restoring needinfo on Ryan. :-)

Comment 34

[Ryan Kelly [:rfkelly]]

So if I had account+sync server set up (account server configured for "autoconfig" (probably with sync-server-url))
I just need to set identity.fxaccounts.autoconfig.uri to the accounts server that would automatically assign the sync server?

Whoops, sorry, I just noticed that I didn't answer this question - yes, that's correct, setting just that one URL would also automatically set up the self-hosted sync server.

Comment 35

[Ryan Kelly [:rfkelly]]

When I set the needinfo? for myself here, I was hoping that we'd follow through on some long-discussed redesigns of how device discovery works in FxA, which might be a good opportunity to revisit the design choices that lead to this surprising user experience. We haven't, and I don't have any reason to think that we're going to in the near term. So, clearing my needinfo? with sadly nothing more to add.