Closed Bug 1351462 Opened 7 years ago Closed 7 years ago

Don't reuse a connection that has not finished an NTLM authentication (may lead to proxy or server confusion, we may open prompt)

Categories

(Core :: Networking, defect)

53 Branch
defect
Not set
normal

Tracking

()

VERIFIED FIXED
mozilla56
Tracking Status
firefox-esr45 --- unaffected
firefox52 --- wontfix
firefox-esr52 - wontfix
firefox53 + wontfix
firefox54 + wontfix
firefox55 + wontfix
firefox56 - fixed

People

(Reporter: raynebc, Assigned: mayhemer)

References

Details

(Whiteboard: [necko-active][ntlm])

Attachments

(1 file, 3 obsolete files)

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170323090023

Steps to reproduce:

User attempts to browse generally any web page containing non local content that has to be fetched by the proxy server.


Actual results:

Firefox prompts for user credentials to use the proxy server instead of using the expected NTLM authentication.  The first Authentication Required box says: “The proxy moz-proxy://[PROXY IP ADDRESS]:[PORT] is requesting a username and password.  It cites “moz-proxy://[PROXY IP ADDRESS]:[PORT]” as the website in question.  When the user cancels this prompt, it is displayed again, only this time it cites [PROXY DNS NAME] as the website.  He is generally able to cancel the prompt and it will then use the proxy server to fetch the web page content.  Occasionally instead it will say the connection is refused and he has to try to load the page again.


Expected results:

Firefox should automatically use NTLM authentication with the proxy server as it used to do prior to Firefox 48.  The user is currently on the beta release channel and has Firefox version 53.0b6.
Honza, do you want to follow up here? This is a followup issue from bug 486508. Maybe we fixed things for most people but still have edge cases. Not sure.
Blocks: 486508
Flags: needinfo?(honzab.moz)
Keywords: regression
Component: Untriaged → Networking
Product: Firefox → Core
Assignee: nobody → honzab.moz
Whiteboard: [necko-active]
Assigned to me.  Will look, but with lower priority.
Flags: needinfo?(honzab.moz)
raynebc, sorry to get to you so late.  If this is reproducible on 53, then it's not bug 1351301.

Can you please find the version that is the first broken?  To make it more easy I suspect that first broken is 52, while 51 probably works.

After you find it, I'd be interested in logs, please see [1] for details how to set them up.

Please modify MOZ_LOG to timestamp,sync,nsHttp:5,negotiateauth:5

Please keep the activity when collecting the log to a minimum (just reproduce the bug and exit).  Perfect would be to have logs from both the working Firefox version and a broken one.

Thanks.
Flags: needinfo?(raynebc)
This behavior (or different bugs with similar resulting prompts for proxy credentials) has been reported by the user all the way back to Firefox 48.  This was the version I documented was installed at the time the user first reported this problem in late August of last year.  Please confirm for which version of Firefox you want us to try to collect logging.  Also please confirm where I get the instructions to achieve your desired log formatting.  I can only guess that the designation of [1] seems to be a footnote/attachment/etc. of some kind but I'm not seeing anything like that on this ticket details page.
Flags: needinfo?(raynebc)
(In reply to raynebc from comment #4)
> This behavior (or different bugs with similar resulting prompts for proxy
> credentials) has been reported by the user all the way back to Firefox 48.

So, this seems like it's not a new regression then.

Is network.automatic-ntlm-auth.allow-proxies and network.negotiate-auth.allow-proxies set to true (the default)?

If not, is the proxy URL listed in the trusted uris preference? (network.automatic-ntlm-auth.trusted-uris, network.negotiate-auth.trusted-uris)

It would be helpful to also test with a clean profile: http://kb.mozillazine.org/Creating_a_new_Firefox_profile_on_Windows
 
> This was the version I documented was installed at the time the user first
> reported this problem in late August of last year.  Please confirm for which
> version of Firefox you want us to try to collect logging.  

I think for now it would be enough to get a log from Firefox 53 (Beta).

> Also please
> confirm where I get the instructions to achieve your desired log formatting.
> I can only guess that the designation of [1] seems to be a
> footnote/attachment/etc. of some kind but I'm not seeing anything like that
> on this ticket details page.

Ah!  I'm sorry.  The link to the logging documentation is at https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Forgot to attach it to the comment.
No longer blocks: 486508
I created a new profile for the user and restored his bookmarks.  I enabled the logging, but we couldn't immediately reproduce the problem so I asked the user to keep playing around with it.  It appears that Firefox creates a new log file for each session, but it's not obvious to me if the log modules that I set in the GUI (about:networking) in one Firefox session (timestamp,sync,nsHttp:5,negotiateauth:5) are kept in future sessions.  When Firefox is re-opened, the "current log modules" field displays the default fields:
timestamp,sync,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5

Mozilla's HTTP logging page (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging) shows the use of an environment variable.  If I define a permanent user environment variable will it keep the desired logging fields for every Firefox session?

If the problem doesn't come back within the next few days, should I have him resume using the old Firefox profile and attempt to get logging for the behavior?
(In reply to raynebc from comment #6)
> I created a new profile for the user and restored his bookmarks.  I enabled
> the logging, but we couldn't immediately reproduce the problem so I asked
> the user to keep playing around with it.  

Then something in the preference settings (about:config) is probably behind the issue.  about:support page content from the profile where you can reproduce the problem would be good to look at.  note it may contain private info.

> It appears that Firefox creates a
> new log file for each session, 

Yes.

> but it's not obvious to me if the log modules
> that I set in the GUI (about:networking) in one Firefox session
> (timestamp,sync,nsHttp:5,negotiateauth:5) are kept in future sessions.  

They are not persisted between sessions, sorry.

> When
> Firefox is re-opened, the "current log modules" field displays the default
> fields:
> timestamp,sync,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
> 
> Mozilla's HTTP logging page
> (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)
> shows the use of an environment variable.  If I define a permanent user
> environment variable will it keep the desired logging fields for every
> Firefox session?

Yes.  It's my preferred way of setting up the logging too.

> 
> If the problem doesn't come back within the next few days, should I have him
> resume using the old Firefox profile and attempt to get logging for the
> behavior?

Yes please.  Note that the logs expose at least URLs that are visited during the session.  Also because the logs are usually large (please zip them before submitting) you can send them directly to my bugzilla email rather than submitting them here as public attachments.

Thanks so far.
raynebc, how is it looking with the logs?
Flags: needinfo?(raynebc)
The user indicated that he hadn't seen the prompt during the past couple of days so I changed it back to using his original Firefox profile, re-enabled the logging and confirmed that it was indicated it was logging the fields specified in the environment variable.  Starting next week, he'll be out of the office the rest of the month, so hopefully if it's going to happen again it will be in the next day and a half.
Flags: needinfo?(raynebc)
The prompts started coming back today.  I'm a bit confused about whether Firefox will create a new log file each time it's opened without manually going into about:networking and starting the logging every single browser session.  The other day it seemed like it would, but I can't find that it reliably does so now.  The user indicated it installed another beta update since yesterday, I don't know if that has anything to do with it.  Is there a way to configure Firefox to permanently enable logging (including for all future browser sessions) until logging is manually turned off?

Luckily the user found a web page that is prone to triggering the proxy authentication behavior for him:
http://hb.511.idaho.gov/#roadReports?timeFrame=TODAY&layers=roadReports%2CwinterDriving%2CweatherWarnings%2CotherStates

Unluckily I can't reproduce this on my Firefox installation by loading the above page, but having that URL might be helpful anyway.  I emailed you the log of a browser session where browsing this page gave him the proxy authentication prompt.
comment 10
Flags: needinfo?(honzab.moz)
(In reply to raynebc from comment #10)
> Is there
> a way to configure Firefox to permanently enable logging (including for all
> future browser sessions) until logging is manually turned off?

You can set the logging.config.clear_on_startup pref to false in about:config, then Start Logging in about:networking
This will keep logging active all the time, but note that this can create huge log files.

Another way would be to start Firefox using these steps: https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging#Windows - this is slightly more complicated.
I'm not interested in Firefox using one giant log file, just a way to ensure logging is started each browser session without having to manually start it each time (or launch Firefox with special steps).  For now I'm guessing it's the Firefox update that had turned off the logging.
(In reply to raynebc from comment #13)
> I'm not interested in Firefox using one giant log file, just a way to ensure
> logging is started each browser session without having to manually start it
> each time (or launch Firefox with special steps).  For now I'm guessing it's
> the Firefox update that had turned off the logging.

Yes, try setting that pref.
I can see the proxy responding to a GET with Type 1 message with a plain 'Proxy-Authenticate: NTLM' [1] and not with the expected Type 2 response.  Hence, this is either the client misconfiguration or the proxy misconfiguration.  We behave correctly according the log.


[1] 2017-04-14 18:19:49.090000 UTC - [Socket Thread]: I/nsHttp   Proxy-Authenticate: NTLM Basic realm="GWNU.<redacted>", transaction 154bc400
Status: UNCONFIRMED → RESOLVED
Closed: 7 years ago
Flags: needinfo?(honzab.moz)
Resolution: --- → INVALID
It didn't use to behave this way with Firefox, and the proxy is working fine with IE and Chrome.  Those browsers must either have more robust handling of different proxy server behavior or Firefox lost such robust handling.  If you're not willing to investigate further, could you at least provide a suitably developer-level technical explanation of the problem that I can send to the company that develops the proxy server we're using?
Status: RESOLVED → VERIFIED
Flags: needinfo?(honzab.moz)
(Nothing has been verified here)

It would be great to also check with the latest Nightly from https://www.mozilla.org/en-US/firefox/channel/desktop/#nightly.  We've made some changes recently around this code.

I will look into the log once more, this could be something with connection handling that makes (surprisingly) the faulty proxy behave unexpectedly.
Status: VERIFIED → RESOLVED
Closed: 7 years ago7 years ago
OK, got it, the proxy is the culprit, but we could workaround this.  Note that this is not a regression but could potentially be more exposed by recent changes.

-----
What happens, all on a single connection:

-> GET /a
<- 407: Proxy-Authenticate: NTLM
-> GET /a Proxy-Authorization: NTLM Type 1
<- 407: Proxy-Authenticate: NTLM Type 2
   the request for /a has been canceled *), the connection is put back to the pool as available and a new request for /b takes it
-> GET /b
<- 407: Proxy-Authenticate: NTLM
-> GET /b Proxy-Authorization: NTLM Type 1
<- 407: Proxy-Authenticate: NTLM
   Firefox handles this as a failure to authenticate with the default creds and opens the prompt

The last response is expected to be 407: Proxy-Authenticate: NTLM Type 2 instead.
-----


A workaround here could be to block reuse of the connection when it's in the middle of the conn-based authentication process.

I think we can DontReuse() the connection when mAuthRetryPending and NS_FAILED(status) and !mAuthConnectionRestartable (=indication of in-progress conn sticky auth) in nsHttpChannel::OnStopRequest.


*) user reloads the page or whatever perfectly legal reason
Assignee: honzab.moz → nobody
Status: RESOLVED → REOPENED
Ever confirmed: true
Flags: needinfo?(honzab.moz)
Keywords: regression
Resolution: INVALID → ---
Summary: Firefox prompts for credentials when using an NTLM authenticating proxy server → Don't reuse a connection that has not finished an NTLM authentication (may lead to proxy or server confusion, we may open prompt)
Whiteboard: [necko-active] → [necko-would-take][ntlm]
Status: REOPENED → NEW
The last few days we've had some more users report usage issues with the latest stable release of Firefox, such as it just ceasing to loads web pages.  In this case it's not asking for credentials like it was for the user referenced throughout this ticket.  You mentioned proxy handling changes in the nightly build, but have such changes made it to the beta update channel?
(In reply to raynebc from comment #19)
> The last few days we've had some more users report usage issues with the
> latest stable release of Firefox, such as it just ceasing to loads web
> pages.  In this case it's not asking for credentials like it was for the
> user referenced throughout this ticket.  You mentioned proxy handling
> changes in the nightly build, but have such changes made it to the beta
> update channel?

Sounds like a different bug from this one.  Is this similar/identical to bug 1361374?  If not, can you file a new bug?  

Please also always try to test with the latest Nightly, we are constantly fixing things around NTLM and they take time to get to the Release channel.  I understand it may not be easy to test with prereleases if reports come externally, tho.
It could also be a regression from bug 1346392 (on 53+) which we are currently fixing in bug 1355858?
It seems different from this bug and the other that you mentioned (bug 1361374).  In this case, no prompt or error is displayed, the page appears to hang and never finish loading.  The user then has to re-open the browser to regain web browsing functionality.

I installed the 55.0a1 nightly on the computer of one of the users affected by this newer issue (silently failing to load pages).  If it can be reproduced I'll probably open a separate ticket, just let me know if any particular logging items are wanted.  If it can't be reproduced in the nightly I guess it would be worth mentioning here in case a future build negatively affects the problem again.
Hi raynebc, the latter may be bug #1360574.
I don't know if the latter issue (silently failing to load pages) is the same as bug #1360574, another affected user just indicated he could get this to occur after loading a half dozen web sites.  When the bug had triggered for him, I opened about:networking but the browser only showed a small handful of HTTP connections and I couldn't see where to find how many TCP connections it had made over the course of that browser session.  I installed the 55.0a1 Firefox nightly on this user's computer and his initial thoughts are that the behavior seemed better, but I asked him to use it for longer and let me know how it goes.

The previous user for whom I installed the nightly (comment #22) has been using Chrome in the mean time but she said she could try to set aside time to test the nightly build a bit more soon.
Per comment #18, the issue comes from the proxy and not a regression. This is a low priority. Mark 54 as fix-optional.
I filed a ticket with the developer of the proxy server we're using.  Regarding the latter issue (silently failing to load pages), if the problem persists for folks even on the nightly build I suppose I'll have to open another ticket.
(In reply to raynebc from comment #26)
> Regarding the latter issue (silently failing to load pages), if the problem
> persists for folks even on the nightly build I suppose I'll have to open
> another ticket.

Yes please, thanks!
Multiple users report that latter issue (silently failing to load pages) still occurs in the nightly build, so I created ticket #1365302.  At this time I don't know if it's nonstandard proxy behavior (like this issue) or an actual Firefox bug.  Would the developer that takes that ticket likely want any specific logging or other initial troubleshooting?
The developers of the Burstek proxy server indicated that this issue is with Firefox instead of their product.  There response to me is as follows:

We have testing the functionality using several browsers and monitored the traffic during authentication and all work properly. Firefox however appears to not wait once it receives the authentication requirement and continues to send URL requests to the proxy. Each request is met with an authentication challenge which is why the login keeps returning. Even when the credentials are supplied, Firefox appears to attempt unauthenticated traffic while the authentication negotiation is occurring. In many cases, Firefox does not even attempt to use the credentials supplied and simply attempts to use either the machine name or the NTAuthority logon. 

We tested with a known working version (50) and the latest (53). Version 50 handles the NTLM challenge response normally. We can see that the workstation and the server are attempting proper authentication however in 53 Firefox appears to be ignoring the responses.
Improper communication on Firefox's side could also explain why some of our power users have recently indicated Firefox is performing much more slowly than it used to and they had to switch over to Chrome.  Personally, I prefer Firefox and would love to recommend it to everybody, but the proxy support seems like it's been fluctuating during the past several months.
Proxy support in Firefox may be getting worse still.  A user pointed out that he couldn't access various usnews.com subdomains (ie. http://travel.usnews.com/ ) with Firefox, as an "Access Denied  You don't have permission to access ... on this server." message would be displayed.  I can reproduce this in Firefox 55.0b3 (32 bit).  The site loads perfectly fine in IE and Chrome when they're using our proxy server, so I don't expect the proxy server is at fault in this instance either.  Should I open a separate bug ticket about this error?
(In reply to raynebc from comment #31)
> Proxy support in Firefox may be getting worse still.  A user pointed out
> that he couldn't access various usnews.com subdomains (ie.
> http://travel.usnews.com/ ) with Firefox, as an "Access Denied  You don't
> have permission to access ... on this server." message would be displayed. 
> I can reproduce this in Firefox 55.0b3 (32 bit).  The site loads perfectly
> fine in IE and Chrome when they're using our proxy server, so I don't expect
> the proxy server is at fault in this instance either.  Should I open a
> separate bug ticket about this error?

If it's a different issue from this bug, could you open a separate bug and provide STR?  Thank you.
See Also: → 1300680
(In reply to Honza Bambas (:mayhemer) from comment #18)
> OK, got it, the proxy is the culprit, but we could workaround this.

In bug #1375579 you affirmed that you tested again and that you believe this bug is due to a problem with the proxy server.  They were certain the problem was with Firefox, but I haven't heard back from Burstek support on this matter in a few weeks so if you wanted to try to convince them, their email address is support@burstek.com.  Otherwise, adding a workaround to Firefox as you suggested would be good because at least then Firefox would achieve the same functional level as IE and Chrome if they are indeed already accounting for the same unexpected proxy behavior.
Assignee: nobody → honzab.moz
Status: NEW → ASSIGNED
Whiteboard: [necko-would-take][ntlm] → [necko-active][ntlm]
ryanbc, if you can, please try testing a build suiting your platform at: https://archive.mozilla.org/pub/firefox/try-builds/honzab.moz@firemni.cz-bc5a228cbe72cb01998cd64015e04d413af208bd/

those are Nightly based builds contain the patch from comment 34.

Thanks.
Flags: needinfo?(raynebc)
Are you wanting me to download firefox-56.0a1.en-US.win32.installer.exe and have the user try that build?  Will this install cleanly alongside the existing Firefox installation like other Nightly builds I've used or does it need to replace the existing installation?
Flags: needinfo?(raynebc)
(In reply to raynebc from comment #36)
> Are you wanting me to download firefox-56.0a1.en-US.win32.installer.exe and
> have the user try that build?  Will this install cleanly alongside the
> existing Firefox installation like other Nightly builds I've used or does it
> need to replace the existing installation?

You can choose destination directory while installing, so you may keep them both installed in your computer.
The profile are shared unless you choose to use another one.
Honza, I installed that nightly build on the user's computer, but he was still able to cause the problem to occur by browsing to the http://forecast.weather.gov website and manipulating the map.  I have attached the logging from one such browser session (log.txt-main 7-6-2017.zip).
(In reply to raynebc from comment #39)
> Honza, I installed that nightly build on the user's computer, but he was
> still able to cause the problem to occur by browsing to the
> http://forecast.weather.gov website and manipulating the map.  I have
> attached the logging from one such browser session (log.txt-main
> 7-6-2017.zip).

Thanks for this test trial.  My patch is incomplete.

Note for me: the problem is that when Cancel on the channel is called before OnStartRequest is received we are not processing the response at all (we never call ProcessResponse() because mStatus == ERROR).  Hence, mAuthRetryPending will never be set and we never execute the parts I've added in the patch.
Comment on attachment 8883347 [details] [diff] [review]
v1

Review of attachment 8883347 [details] [diff] [review]:
-----------------------------------------------------------------

cancel r? based on comment 40
Attachment #8883347 - Flags: review?(mcmanus)
Attachment #8885780 - Attachment is obsolete: true
Attachment #8885780 - Flags: review?(mcmanus)
ryanbc, can you please retest again with a build from https://archive.mozilla.org/pub/firefox/try-builds/honzab.moz@firemni.cz-0e68e244367d586d4a1699dde5404f9d2606256c/ ?

Thanks.
 
(will take couple hours for them to appear on the server)
Flags: needinfo?(raynebc)
Comment on attachment 8885783 [details] [diff] [review]
v2

Review of attachment 8885783 [details] [diff] [review]:
-----------------------------------------------------------------

seems sensible enough...
Attachment #8885783 - Flags: review?(mcmanus) → review+
Honza, the user has not been able to reproduce the problem so far with your nightly build from comment #44.  I'll let you know if he indicates otherwise to me, otherwise the problem may be fixed.
Flags: needinfo?(raynebc)
(In reply to raynebc from comment #46)
> Honza, the user has not been able to reproduce the problem so far with your
> nightly build from comment #44.  I'll let you know if he indicates otherwise
> to me, otherwise the problem may be fixed.

Thanks for the confirmation!  I'm going to land this now.
Checked that this applies cleanly on current m-c (presuming m-i as well, not aware of many changes happening these days in this code)
Keywords: checkin-needed
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/151b602f0649
Don't reuse a connection that has not finished an NTLM authentication. r=mcmanus
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/151b602f0649
Status: ASSIGNED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Please request Beta & ESR52 approval on this when you get a chance.
Flags: needinfo?(honzab.moz)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #51)
> Please request Beta & ESR52 approval on this when you get a chance.

Let this ride - sensitive and not a sec issue.  And in this case it's about a faulty proxy server, we are only working around a non-necko issue.
Flags: needinfo?(honzab.moz)
I couldn't tell from your comment, is this fix/workaround going to become part of the official Firefox 56 branch?
Yes, just not getting backported to 55 or ESR52.
Honza:  The user reported that the authentication prompts just started coming back.  He indicated he currently has "56.0a1 (2017-07-12)" the date for which lines up with that nightly you previously asked us to try.  I'm guessing it hasn't automatically updated to another build on its own?  I'm not sure why the problem would take until now to occur, do you want me to collect and submit the usual logging?
(In reply to raynebc from comment #55)
> Honza:  The user reported that the authentication prompts just started
> coming back.  He indicated he currently has "56.0a1 (2017-07-12)" the date
> for which lines up with that nightly you previously asked us to try.  I'm
> guessing it hasn't automatically updated to another build on its own?  I'm
> not sure why the problem would take until now to occur, do you want me to
> collect and submit the usual logging?

Please open a new bug with (few more details, if available) and yes, a log would be definitely useful to have.  Also please include content of about:buildconfig page of that user to make sure she/he's running a version containing this fix.  

Thanks!
(In reply to Honza Bambas (:mayhemer) from comment #56)
> (In reply to raynebc from comment #55)
> > Honza:  The user reported that the authentication prompts just started
> > coming back.  He indicated he currently has "56.0a1 (2017-07-12)" 

Wait...  I had to read more carefully...  This has been fixed in 56 on 2017-07-26 (comment 50).  Hence, that version he's using doesn't contain the fix.  Yes, he should update to the latest Nightly or Beta, which has the fix.
How do I switch him off of your nightly build?  Do I have the About Firefox dialog look for updates or do I need to manually install the latest nightly release?
(In reply to raynebc from comment #58)
> How do I switch him off of your nightly build?  Do I have the About Firefox
> dialog look for updates or do I need to manually install the latest nightly
> release?

The safest and most reliable is to simply download and run the installer again.  It also allows you to change update channel (Beta or Release) if wanted.

Note that Beta has the fix now.
Do you recommend we install the latest offered beta version instead of nightly?
The user indicated he hasn't encountered any unexpected authentication prompts since we installed Firefox 56b4 on his computer about 3 weeks ago.
(In reply to raynebc from comment #61)
> The user indicated he hasn't encountered any unexpected authentication
> prompts since we installed Firefox 56b4 on his computer about 3 weeks ago.

Thanks for confirming it!
Status: RESOLVED → VERIFIED
Depends on: 1451293
You need to log in before you can comment on or make changes to this bug.