1219219 - Firefox Sync fails behind Proxy-servers/Proxy-authencation

Status: UNCONFIRMED

(Firefox :: Sync, defect, P4)

Description

[stefan.mueller.83]

Attachment: error-sync-1445238680985.txt

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0
Build ID: 20151014143721

Steps to reproduce:

Hallo,
it seems the outcome of my problem is quite similar to what is described in [4] although the root cause is properly different.
Here it comes:
My company's network as been recently upgraded with the Zscaler proxy solution. I was able to regain internet access by following that thread [1]. Unfortunately that dind't worked out for Firefox Sync. I'm able to log-in (verified by getting a mail from Firefox Accounts) but Firefox isn't stil able to sycn. I suspect that is somehow related to the certification signing process/getting token8gaining access what is properly blocked by zscaler.
Before FF 29, it seems it was possible to solve that issue see here [2].

BTW. I've already started a thread on mozilla support [3] for furher details


[1] https://superuser.com/questions/115349/firefox-this-connection-is-untrusted-behind-corporate-firewall
[2] http://www.discourse.net/2010/12/how-to-solve-the-dreaded-firefox-sync-error-while-signing-in-problem/#comment-1358006
[3] https://support.mozilla.org/de/questions/1088686#answer-796661
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1118958


Actual results:

Firefox Sync wasn't successful 


Expected results:

proper sync process

Comment 1

[Mark Hammond [:markh] [:mhammond]]

The logs show that our post to our /certificate/sign endpoint is failing with:

> 445238634658	FirefoxAccounts	ERROR	error POSTing /certificate/sign: {"code":401,"errno":109,"error":"Unauthorized","message":"Missing authentication","info":"https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md#response-format"}

That request uses an "Authorization" header to send the authentication info, so it looks alot like the proxy is stripping or modifying that header. I'm not familiar with zscaler so I can't speculate why that would be.  CC :rfkelly who might have more insight.

Comment 2

[Ryan Kelly [:rfkelly]]

> so it looks alot like the proxy is stripping or modifying that header

Agreed, this seems like the most likely explanation.  The solution noted in [2] seems to be about accepting the zscaler certificate, which it sounds like you've already done if you're able to successfully create the account.

Unfortunately I'm not able to find much info about why or how zscaler might trigger this behaviour.  It might e.g. block requests with "non-standard" Authentication headers like the Hawk-based header we use here.

Does you company happen to have a support agreement with zscaler?  The best way to move this forward may be to get their support people involved to answer some questions about the expected handling of such requests.

Comment 3

[stefan.mueller.83]

hallo
sorry for ma late reply.

Normal internet access is working fine I can even access & edit my Firefox Sync account.
It seems it fails only when Firefox when the token handshake should takes place.

I don't see a big chance to get Zscaler support involved but I'll try my best.

Comment 4

[stefan.mueller.83]

That might be little hint 
I'm able to login into and use pocket via my firefox (sync) account

Comment 5

[stefan.mueller.83]

FF 45 shows some more information what failed, may that helps a bit.
Would be nice that it shows up the login-in/authentication page for the proxy as for all other https sites.
Fehlercode: SEC_ERROR_REUSED_ISSUER_AND_SERIAL

Comment 6

[jeremy.philippe]

Hello,

I've been struggling with this issue for a long time (FF sync not working through ZScaler proxy) and I've finally made it work by trying to add a certificate exception for https://accounts.firefox.com (Options->Advanced->Certificates->View Certificates->Add Exception).

Then I got the message "certificate is valid, no need to add an exception", and for some reason the manual probing now means FF sync is now working...

Maybe this might help someone else?

Comment 7

[Chris Karlof [:ckarlof]]

Getting a HTTP log including the headers might be helpful here.

Comment 8

[stefan.mueller.83]

would like to do so, if you tell me how

Comment 9

[Chris Karlof [:ckarlof]]

Redirecting to Mark.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Chris, can you please answer Stefan's question on how to get the logs you're wanting?

Comment 11

[Mark Hammond [:markh] [:mhammond]]

(In reply to stefan.mueller.83 from comment #8)
> would like to do so, if you tell me how

Thanks for offering to help. To be honest, getting this data will require some understanding of networking, so I'll understand if this turns out to be an unreasonable request.

The best way is probably to use a tool such as https://www.wireshark.org/ or the Microsoft Network Monitor (https://www.microsoft.com/en-us/download/details.aspx?id=4865), then attempting to perform a Sync and attaching the captured data to this bug (or if you prefer due to potentially sensitive data, email it to me directly).

Another alternative would be to use the Firefox "Browser Toolbox" which has the ability to see the requests and responses made by the browser, and copy/paste the request and response that is failing due to the proxy.

I hope this helps - please feel free to ask for more information if you need it.

Comment 12

[stefan.mueller.83]

wireshark.org needs WinPcap what can be installed only with admin rights, same as Microsoft Network Monitor.
So I tried Firefox "Browser Toolbox". I reckon you want me to do an network analysis by the means of Network Monitor. That's easy but I have no clue how to export all the log entries same for Firebug.
There's isn't any export button and "Save all as Har file" doesn't save anything in my profile folder(besides that is only for a single entry.
BTW, it's hard to find any applicable resource regarding the export of result generated by the Network Monitor.

Comment 13

[Mark Hammond [:markh] [:mhammond]]

I tried "Copy all as HAR" and it put some JSON into my clipboard - you could save that to a file and attach it.

Comment 14

[stefan.mueller.83]

As far as I understand that only copies the selected "http log" / JSON into the clipboard but Net Monitor lists a lot of "http get request" etc and I don't know which do copy/store. It will take too much time to copy all of them and sort them properly.

Comment 15

[stefan.mueller.83]

Attachment: Network Monitor Log.png

here is a screenshot of Network Monitor Log when pressing on sign-in button in settings->sync

Comment 16

[Mark Hammond [:markh] [:mhammond]]

(In reply to stefan.mueller.83 from comment #15)
> here is a screenshot of Network Monitor Log when pressing on sign-in button
> in settings->sync

Thanks - but that is odd in that it seems to show the post to certificate/sign actually worked. When I'm expecting to see is a request that fails with a 401. Were there any requests after that?

Comment 17

[stefan.mueller.83]

Attachment: firefox sycn.7z

that's all what is happening

Comment 18

[Mark Hammond [:markh] [:mhammond]]

I'll look at the logs tomorrow - it is the "POST" to "sign" that the earlier logs said were failing - it would be great if you could try and extract as much request and response information from that and attach it here.

Comment 19

[Jean-Luc B]

FF Sync does not work at all behind Zscaler proxy despite a perfectly functional web browsing.

Comment 20

[stefan.mueller.83]

Attachment: Network Monitor - entire log.txt

save as HAR saves the entire log :)

Comment 21

[Mark Hammond [:markh] [:mhammond]]

Ah, I think this is from the login page only - but that isn't showing the requests Sync itself is making. What I need to see is all the requests from the "Browser Toolbox". Can you please follow the instructions at https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox to enable the toolbox and save a HAR file for all requests made by the browser?

Comment 22

[stefan.mueller.83]

Attachment: firebug log - from brower toolbox - #1 .txt

Comment 23

[stefan.mueller.83]

Attachment: firebug log - from brower toolbox - #2.txt

Comment 24

[stefan.mueller.83]

Attachment: Network Monitor - from browser toolbox - entire log #1.txt

Comment 25

[stefan.mueller.83]

Attachment: Network Monitor - from browser toolbox - entire log #2.txt

Comment 26

[stefan.mueller.83]

Browser Toolbox was active before but this time I opened the box (Ctrl+Shift+Alt+I) and took the logs from that network Monitor, hope it has now all information

Comment 27

[Mark Hammond [:markh] [:mhammond]]

Awesome - I think I know what is going on.

This is happening on a number of requests, but here is one example:

A request is made to fetch a token so we can start syncing:
{
        "request": {
          "bodySize": 0,
          "method": "GET",
          "url": "https://token.services.mozilla.com/1.0/sync/1.5",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Host",
              "value": "token.services.mozilla.com"
            },
            ...
            {
              "name": "Authorization",
              "value": "BrowserID ...",
            }
        },
        "response": {
          "status": 307,
          "statusText": "Temporary Redirect",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Content-Length",
              "value": "0"
            },
            {
              "name": "Location",
              "value": "https://gateway.zscaler.net:443/auD?origurl=https%3A%2F%2Ftoken%2eservices%2emozilla%2ecom%2f1%2e0%2fsync%2f1%2e5&_ordtok=7Z43WV5VFLrMr7DLMR8QJR4NJ9"
            },
            ...
        }
}

So this request sees a redirection to the zscaler gateway. However, when we follow that redirection we *do not* copy the Authorization header in the redirected request due to the logic at https://dxr.mozilla.org/mozilla-central/rev/3e8ee3599a67edd971770af4982ad4b0fe77f073/services/common/rest.js#596, which plays safe and refuses to copy the auth header. As a result the redirected request fails.

The restrictions on when we copy the headers was added in bug 798430, and as bug 798430 comment 15 states, there is a "significant security risk" in just blindly allowing those headers through for all redirects. Brian, that comment is yours :) Do you have any insights into how we should handle proxies here?

Comment 28

[Patrick McManus [:mcmanus]]

This smells like a captive portal login (or perhaps just verification) to me. after we follow the redirect (without the Authorization header that is meant for sync, not zscaler - I think what brian wrote about that is correct) what do we see then? I'm wondering if it eventually ends up in another redirect to sync and _that's_ the request that needs the auth token.

Comment 29

[Mark Hammond [:markh] [:mhammond]]

Best I can tell we see:

GET https://token.services.mozilla.com/1.0/sync/1.5 -> 307 https://gateway.zscaler.net:443/auD?origurl=https%3A%2F%2Ftoken%2eservices%2emozilla%2ecom%2f1%2e0%2fsync%2f1%2e5&_ordtok=mV43WVqqmWK0DlVNqzzJ56mn6N

So then we follow the redirect:

{
        "pageref": "page_1",
        "startedDateTime": "2016-06-06T13:34:56.373+02:00",
        "time": 45,
        "request": {
          "bodySize": 0,
          "method": "GET",
          "url": "https://gateway.zscaler.net/auD?origurl=https%3A%2F%2Ftoken%2eservices%2emozilla%2ecom%2f1%2e0%2fsync%2f1%2e5&_ordtok=mV43WVqqmWK0DlVNqzzJ56mn6N",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Host",
              "value": "gateway.zscaler.net"
            },
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:46.0) Gecko/20100101 Firefox/46.0"
            },
            {
              "name": "Accept",
              "value": "application/json"
            },
            {
              "name": "Accept-Language",
              "value": "de,en-US;q=0.7,en;q=0.3"
            },
            {
              "name": "Accept-Encoding",
              "value": "gzip, deflate, br"
            },
            {
              "name": "Connection",
              "value": "keep-alive"
            },
            {
              "name": "Pragma",
              "value": "no-cache"
            },
            {
              "name": "Cache-Control",
              "value": "no-cache"
            }
          ],
          "cookies": [],
          "queryString": [
            {
              "name": "origurl",
              "value": "https://token.services.mozilla.com/1.0/sync/1.5"
            },
            {
              "name": "_ordtok",
              "value": "mV43WVqqmWK0DlVNqzzJ56mn6N"
            }
          ],
          "postData": {
            "mimeType": "",
            "params": [],
            "text": ""
          },
          "headersSize": 423
        },
        "response": {
          "status": 200,
          "statusText": "Connection Established",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Content-Length",
              "value": "0"
            },
            {
              "name": "Location",
              "value": "https://gateway.zscaler.net:443/auT?origurl=https%3A%2F%2Ftoken%2eservices%2emozilla%2ecom%2f1%2e0%2fsync%2f1%2e5&_ordtok=mV43WVqqmWK0DlVNqzzJ56mn6N"
            },
            {
              "name": "Content-Type",
              "value": "text/html"
            },
            {
              "name": "Set-Cookie",
              "value": "_sm_au_d=1;path=/;domain=gateway.zscaler.net;Secure;HttpOnly;"
            },
            {
              "name": "P3P",
              "value": "CP=\"NOI ADM DEV PSAi COM NAV OUR OTR STP IND DEM\""
            }
          ],
          "cookies": [
            {
              "name": "_sm_au_d",
              "value": "1"
            }
          ],
          "content": {
            "mimeType": "application/json; charset=UTF-8",
            "size": 96,
            "text": "{\"status\": \"error\", \"errors\": [{\"location\": \"body\", \"name\": \"\", \"description\": \"Unauthorized\"}]}"
          },
          "redirectURL": "https://gateway.zscaler.net:443/auT?origurl=https%3A%2F%2Ftoken%2eservices%2emozilla%2ecom%2f1%2e0%2fsync%2f1%2e5&_ordtok=mV43WVqqmWK0DlVNqzzJ56mn6N",
          "headersSize": 65,
          "bodySize": 96
        },

The "content" above is from our server (ie, it's the same response as when you paste https://token.services.mozilla.com/1.0/sync/1.5 into the URL bar), so the proxy appears to have hit the original URL - but obviously without the Auth header. That response does have a Location header, but still pointing at https://gateway.zscaler.net:443... - best I can tell there's nothing in that response that indicates we should redirect back to the original URL.

Patrick, does that answer your question, or give you any further clues as to what is going on here?

Comment 30

[Mark Hammond [:markh] [:mhammond]]

Also, the original 307 response attempts to set a cookie (which IIUC should then be used for token.services.mozilla.com). Then the redirected response also sets a cookie (which will be applied to zscaler.net).

(In reply to Mark Hammond [:markh] from comment #29)
> the Auth header. That response does have a Location header, but still
> pointing at https://gateway.zscaler.net:443...

I wonder if the proxy is expecting us to ignore the response here, but follow the redirect while supplying that second cookie? Doing that may end up redirecting back to the original domain, where we'd send the first cookie, and maybe we'd then get the response we are after. I can't imagine why they would otherwise be offering those cookies...

</me inserts the "I have no idea what I'm doing" meme...>

Comment 31

[Patrick McManus [:mcmanus]]

I know you're using the term proxy as a logical term, but what we're seeing isn't an HTTP proxy in any sense - HTTP proxies do not change the origin of their resources via redirects and location headers.. consider cookies in this approach, you certainly wouldn't consider sending your mozilla.com cookies to a zscaler origin, right? This is why captive portals normally do the double redirect.. once to get you to the CP and then they redirect you back to the triggering page so the UA will insert the correct auth. I can't tell for sure from these logs whether or not firefox thinks it is using an http proxy - https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging - would say for sure, but I'm pretty sure its not doing so.

I wonder if the report is just based on a misconfiguration - does the enterprise provide a proxy address that should be configured into firefox, and this redirection layer is just what happens if that is not done giving a sort-of working experience that makes us think firefox is the issue.

I also wonder if it does this redirect jazz for every single transaction? That would barely work with large POSTs (if at all).

what does happen if https://token.services.mozilla.com/1.0/sync/1.5 is pasted into the reporter's url bar? If we get the same kind of redirection the location bar would actually change to reflect zscaler - but again, that's not how proxies normally work (the whole web would suddenly live under 1 origin - gateway.zscaler.net, which would make a mockery of single origin security policies, etc..)

Anyhow it does look like this would work if you just copied the auth header, but that seems like the wrong thing to do. I think the first step should be double checking to see if this zscaler install really requires a proxy install in the client that hasn't been done. Failing that, I would be happy to take a call with zscaler to try and understand what they are expecting if anyone wants to set it up.

Comment 32

[stefan.mueller.83]

when I open https://token.services.mozilla.com/1.0/sync/1.5 I get:
     Secure Connection Failed   
     An error occurred during a connection to oken.services.mozilla.com.
     You have received an invalid certificate.  Please contact the server administrator or email
     correspondent and give them the following information:
     Your certificate contains the same serial number as another certificate issued by the certificate
     authority. Please get a new certificate containing a unique serial number.
     (Error code: sec_error_reused_issuer_and_serial)
     ...

(There is nothing in site information.)

Normally when I open pages on a secure connection the very first time I get an login request, where I have to put in my windows credentials. Afterwards I will be redirected to the requested site.

Comment 33

[Mark Hammond [:markh] [:mhammond]]

Stefan,
  I'm afraid that I don't see anything further we can do without help from ZScaler, so assuming your org is a paying customer, the best thing to do is to open a support ticket with ZScaler and point them at this bug.

Comment 34

[stefan.mueller.83]

I conntaced Zscaler but the only answer I got isn't that helpful:
   As discussed on the 27th comment, Mozilla does not return the Authorization Header in the
   redirected request from the Zscaler due to some security risk.
   The only option is to Auth bypass the URL in the Zscaler Admin portal.

Isn't somehow possible to the login/token exchange graphical, so that I all connection/data exchange/redirect can be authorised as you do when using https connections.

Comment 35

[Mark Hammond [:markh] [:mhammond]]

(In reply to stefan.mueller.83 from comment #34)
> I conntaced Zscaler but the only answer I got isn't that helpful:
>    As discussed on the 27th comment, Mozilla does not return the
> Authorization Header in the
>    redirected request from the Zscaler due to some security risk.

Heh - "some security risk" is understating the problem. An authorization header is defined as being specific to a single "realm" and may contain a (basically) clear-text copy of the user's password for the initial server. If we sent that header on a redirect to a different domain then (a) it is not valid for that domain as it is a different realm, and (b) that redirected server now knows the username and password for a completely different server.

While Sync is not sending the password in that way, I hope it's fairly clear that this would be a significant risk in the general case.

>    The only option is to Auth bypass the URL in the Zscaler Admin portal.

That sounds like it might work for you?

> Isn't somehow possible to the login/token exchange graphical, so that I all
> connection/data exchange/redirect can be authorised as you do when using
> https connections.

I'm not sure what you are saying here, but it would be reasonable to argue the auth header *is* sent in the case of a redirection in the same domain (eg, a redirect from http -> https) but I can't see any justification for sending that header when the redirection is to an unrelated domain - as per RFC 2616, that header can't possibly apply to that redirected domain and as mentioned above, may leak the username and password in the general case.

Comment 36

[stefan.mueller.83]

>>The only option is to Auth bypass the URL in the Zscaler Admin portal.

>That sounds like it might work for you?

Unfortunately not, I even set connection settings to "no proxy" but when I initialize a https connection I still see Zscaler working in the background (in the status bar, where FF tells me what is currently loading). It seems that what ever is in the connection settings, the Zscaler proxy is always involved.  

>> Isn't somehow possible to the login/token exchange graphical, so that I all ...

> I'm not sure what you are saying here,...
Some kind of website based login process for the sync service "analogue" to login in into https://www.accounts.firefox.com.
Doing so the user is may allowed to accept that header will be send to an unrelated domain. For sure is has to be clearly shown what information will be sent to what domain, instead of...Something similar, when Firefox warns you if a site wants to redirect or refresh.

Comment 37

[Mark Hammond [:markh] [:mhammond]]

(In reply to stefan.mueller.83 from comment #36)
> >>The only option is to Auth bypass the URL in the Zscaler Admin portal.
> 
> >That sounds like it might work for you?
> 
> Unfortunately not, I even set connection settings to "no proxy" but when I
> initialize a https connection I still see Zscaler working in the background
> (in the status bar, where FF tells me what is currently loading). It seems
> that what ever is in the connection settings, the Zscaler proxy is always
> involved. 

You should probably ask ZScaler support for help with that.
 
> > I'm not sure what you are saying here,...
> Some kind of website based login process for the sync service "analogue" to
> login in into https://www.accounts.firefox.com.
> Doing so the user is may allowed to accept that header will be send to an
> unrelated domain. For sure is has to be clearly shown what information will
> be sent to what domain, instead of...Something similar, when Firefox warns
> you if a site wants to redirect or refresh.

That's not really viable given how Sync authentication works, and even if it was, it would be alot of effort to work around what is clearly a broken proxy solution.

Comment 38

[stefan.mueller.83]

Great news,
can't tell what happened but at least as from FF 54 sync is working may even in FF 53.
Do you want to have the log file to figure out why it's now working.

Comment 40

[PierU]

I come from the bug 1503202, and have strange new observations...

Reminder : Sync doesn't work on my company's laptop, when it is on the company's network behind the zscaler proxy (constantly asking to "reconnect to sync"). The problem apparently started 15 days ago, maybe with a configuration change on zcaler side. And Sync does work when the laptop is at home, without the proxy. So it looks like something bad happens to Sync with zscaler. However : 
- At home I tried to set the zscaler proxy and see what happens : Sync is working !
- I then tried to enable the company's VPN : Sync is still working... 

With both the proxy and the VPN on, it should be as if the laptop was on the company's network, so I was expecting Sync not to work in this situation. So I don't understand what happens... unless Sync be no longer blocked by zscaler. I will be able to check that at the office, next monday.

Comment 41

[colin.huckstep]

I see a similar issue with Zscaler. Specifically with the PAC file running on a VDI, using per session SAML auth with ADFS. On my laptop using the Zscaler App I don't have issues. I suspect what is happening is that the Sync Process is spawning a new Session in the background, which is not using the settings from network.automatic-ntlm-auth.trusted-uris to automatically sign me in to Zscaler as my user that is logged into windows.

Until I put in my trusted domains for network.automatic-ntlm-auth.trusted-uris I am prompted for credentials every time I start a new browser session.

(In reply to PierU from comment #40)
> I come from the bug 1503202, and have strange new observations...
> 
> Reminder : Sync doesn't work on my company's laptop, when it is on the
> company's network behind the zscaler proxy (constantly asking to "reconnect
> to sync"). The problem apparently started 15 days ago, maybe with a
> configuration change on zcaler side. And Sync does work when the laptop is
> at home, without the proxy. So it looks like something bad happens to Sync
> with zscaler. However : 
> - At home I tried to set the zscaler proxy and see what happens : Sync is
> working !
> - I then tried to enable the company's VPN : Sync is still working... 
> 
> With both the proxy and the VPN on, it should be as if the laptop was on the
> company's network, so I was expecting Sync not to work in this situation. So
> I don't understand what happens... unless Sync be no longer blocked by
> zscaler. I will be able to check that at the office, next monday.

Comment 42

[Mark Hammond [:markh] [:mhammond]]

(In reply to colin.huckstep from comment #41)
> I see a similar issue with Zscaler. Specifically with the PAC file running
> on a VDI, using per session SAML auth with ADFS. On my laptop using the
> Zscaler App I don't have issues. I suspect what is happening is that the
> Sync Process is spawning a new Session in the background, which is not using
> the settings from network.automatic-ntlm-auth.trusted-uris to automatically
> sign me in to Zscaler as my user that is logged into windows.

I'm not sure what a "Session" means in this context, but Sync happens inside the Firefox process and uses the fetch() API to hit the network, so I don't understand how what we do is any different to any other network request made by Firefox.

Comment 45

[PierU]

Wow, the sync has suddenly started to work behind our zcaler proxy... Could it be because our admin has updated FF to 60.4.0esr (from 60.3.0esr) ?

Comment 46

[PierU]

Gosh, it stopped working again :-(

Since there has been no FF update, the problem is clearly on the proxy side (or in the script to access the proxy)...

Comment 47

[PierU]

Fresh news : sync is working behind our zscaler proxy since the FF update to version 68 ESR (1 month ago). Coincidence ? We'll see...

Comment 49

[PierU]

On my side it has been correctly working for 2 years now. No more issue between Sync and our proxy.

Comment 50

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 4 duplicates.
:skhamis, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 51

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.