Last Comment Bug 785050 - ISA proxy negotiate auth broken (regression)
: ISA proxy negotiate auth broken (regression)
Status: RESOLVED FIXED
: regression
Product: Core
Classification: Components
Component: Networking (show other bugs)
: Trunk
: x86 Windows XP
: -- normal with 1 vote (vote)
: mozilla19
Assigned To: Patrick McManus [:mcmanus]
:
Mentors:
: 804574 (view as bug list)
Depends on:
Blocks: 767158
  Show dependency treegraph
 
Reported: 2012-08-23 06:15 PDT by will69
Modified: 2013-04-03 11:04 PDT (History)
9 users (show)
ryanvm: in‑testsuite?
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
unaffected
+
unaffected
+
unaffected
fixed


Attachments
patch 0 (1.41 KB, patch)
2012-10-16 10:18 PDT, Patrick McManus [:mcmanus]
jduell.mcbugs: review+
cbiesinger: review+
akeybl: approval‑mozilla‑aurora+
akeybl: approval‑mozilla‑beta+
Details | Diff | Review

Description will69 2012-08-23 06:15:06 PDT
Requesting any web page yields HTTP error 407 (proxy authentication required) and status 12209 (The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied).

STR: Use a MS ISA server as your proxy with a default configuration of Firefox 17.

Workaround: Set "network.negotiate-auth.allow-proxies" from "true" to "false" in "about:config" (just a double click)

Regression range:

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=20db7c6d82cc&tochange=8b96a33ecbd2

Download:

GOOD: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/07/2012-07-26-05-28-03-mozilla-central/firefox-17.0a1.en-US.win32.installer.exe

FAIL: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/07/2012-07-27-03-05-08-mozilla-central/firefox-17.0a1.en-US.win32.installer.exe
Comment 1 Matthias Versen [:Matti] 2012-08-23 10:21:21 PDT
Thank you for the excellent bug report with a regression range !
I suspect bug 767158
Comment 2 Patrick McManus [:mcmanus] 2012-09-06 17:57:00 PDT
will69 - do you know what kind of authorization your proxy is doing? Specifically

1] do you know if it is NTLM or Negotiate?

2] is it full windows integrated auth? Do you normally have to type in your credentials to authenticate to the proxy or is it (before this bug) transparent to you if you are logged in?

thanks
Comment 3 Patrick McManus [:mcmanus] 2012-09-06 18:07:33 PDT
3] (probably redundant, but what the heck) is your windows computer attached to a domain controller that is being used to determine the auth for the ISA proxy?
Comment 4 will69 2012-09-07 12:28:07 PDT
1] Iā€™m afraid I need some help to find out. Is there a plugin I can use?
2] It is fully integrated. Before this bug, I did not have to type in my credentials.
3] The computer is a member of one domain. The user is a member of another domain. Both are in the same forest. Access to the proxy is granted to the user via a security group.
Comment 5 Patrick McManus [:mcmanus] 2012-10-11 12:24:16 PDT
will I'm afraid this is works for me. I've setup a windows ADC and I've setup a copy of wingate that uses windows integrated autthentication (negotiate::kerberos) from the ADC, and I've added my laptop to that domain..

and I can confirm that the auth is enforced and works in nightly, ff15, and msie all the same.

not sure where to go here.. can you do a packet trace (see wireshark) with both nightly and ff15 or ff16 so we can see the differences?
Comment 6 michael-mozilla 2012-10-14 21:56:11 PDT
I think I've run into this on ff17 beta, and having just downgraded to 16.0.1, I'm sure of it.

Let me know if you need anything else for debugging purposes.  network.*auth* is set to default, except for network.negotiate-auth.trusted-uris, which is set to a domain specific value I can't really disclose.  However, unsetting network.negotiate-auth.trusted-uris doesn't do anything.

I do have FoxyProxy installed, but disabling it doesn't improve matters. 

From wireshark, my initial (working, ff16.0.1) trace says:

CONNECT bugzilla.mozilla.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0
Proxy-Connection: keep-alive
Host: bugzilla.mozilla.org

HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
Via: 1.1 XXXXXXXXXXX
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Connection: close
Proxy-Connection: close
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 737

Which is followed by:

CONNECT bugzilla.mozilla.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0
Proxy-Connection: keep-alive
Host: bugzilla.mozilla.org
Proxy-Authorization: Negotiate TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAFASgKAAAADw==

HTTP/1.1 407 Proxy Authentication Required ( Access is denied.  )
Via: 1.1 XXXXXXXXXXXXX
Proxy-Authenticate: Negotiate TlRMTVNTUAACAAAADgAOADgAAAAFgomib3t0+h9SBh0AAAAAAAAAAMQAxABGAAAABQLODgAAAA9PAEMARQBBAE4ASQBBAAIADgBPAEMARQBBAE4ASQBBAAEAGgBQAFgAWQBOAFoAMAA2ADEATABEAEEAMAAwAAQAKABvAGMAZQBhAG4AaQBhAC4AYwBvAHIAcAAuAGEAbgB6AC4AYwBvAG0AAwBEAFAAWABZAE4AWgAwADYAMQBMAEQAQQAwADAALgBvAGMAZQBhAG4AaQBhAC4AYwBvAHIAcAAuAGEAbgB6AC4AYwBvAG0ABQAYAGMAbwByAHAALgBhAG4AegAuAGMAbwBtAAAAAAA=
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 0     

CONNECT bugzilla.mozilla.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0
Proxy-Connection: keep-alive
Host: bugzilla.mozilla.org
Proxy-Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAIAAAAAYABgAmAAAAA4ADgBIAAAAEAAQAFYAAAAaABoAZgAAAAAAAACwAAAABYKIogUBKAoAAAAPTwBDAEUAQQBOAEkAQQByAG8AYgBpAG4AcwBtADUAVwA1AEMAMgA2ADAAQQA0ADQAMQAyADkARQBdpiGV1aa44gAAAAAAAAAAAAAAAAAAAABcYnHGzWxuf2pjeRA5bWVzd/pbSh8Lkbw=

HTTP/1.1 200 Connection established
Via: 1.1 XXXXXXXXXXXXX, 1.1 XXXXXXXXXXXXX
Connection: Keep-Alive
Proxy-Connection: Keep-Alive

And then a non-working version (17.0b1), using the same profile:

CONNECT bugzilla.mozilla.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/17.0 Firefox/17.0
Proxy-Connection: keep-alive
Host: bugzilla.mozilla.org

HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
Via: 1.1 XXXXXXXXXXXXX
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Connection: close
Proxy-Connection: close
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 737   

Followed by, well, nothing.
Comment 7 Patrick McManus [:mcmanus] 2012-10-15 11:27:21 PDT
this is still wfm, this time using TMG 2010 which is the successor to ISA.

Michael-Mozilla, it looks like you've got a double depth proxy of some sort based on 

HTTP/1.1 200 Connection established
Via: 1.1 XXXXXXXXXXXXX, 1.1 XXXXXXXXXXXXX

any chance you can test with just 1 proxy (I know maybe you can't).

will69, any chance you've got 2 layers of proxies too?
Comment 8 Patrick McManus [:mcmanus] 2012-10-15 11:44:19 PDT
Michael-Mozilla,

Since this remains worksForMe, can you make an HTTP log for me with the failing scenario?

It should be similar to https://developer.mozilla.org/en-US/docs/HTTP_Logging except the NSPR_LOG_MODULES like should contain negotiateauth:5

i.e.

set NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,nsHostResolver:5,=negotiateauth:5

and set NSPR_LOG_FILE to something which you will upload.

Feel free to read it and redact if 100% necessary - or send it to me privately (pmcmanus@mozilla.com) if that works better. If you do redact please point those out for me.

hopefully with that we can figure out why it isn't sending the request with the Proxy-Authorization that gets challenged.
Comment 9 michael-mozilla 2012-10-15 13:51:33 PDT
Patrick,

Have sent you the nspr log privately.  Unfortunately I can't test with just one proxy; I have no control over my local bureaucracies' proxies :(.
Comment 10 will69 2012-10-16 03:09:52 PDT
Patrick,

we do indeed have a proxy hierarchy. However, web browsers are configured to access Microsoft ISA server 2006 SP1 running on Microsoft Windows Server 2003 with integrated NLB (1 virtual IP, 3 servers). ISA is configured to do "integrated" auth.

I sent you an NSPR log privately, too. (Cannot easily do a WS scan.)
Comment 11 Patrick McManus [:mcmanus] 2012-10-16 07:46:27 PDT
micheal-mozilla, will69,

do you guys have "broken" dns at your desktop? i.e. most DNS is handled by the proxy and your desktop can only successfully resolve a few local names?

that would be consistent with the log and would explain why I can't reproduce. (I do have proxies setup but I also have working internet connectivity).
Comment 12 will69 2012-10-16 07:59:51 PDT
Yes, DNS traffic is restricted: Resolving internet hosts is not possible.
Comment 13 Patrick McManus [:mcmanus] 2012-10-16 08:01:15 PDT
(In reply to will69 from comment #12)
> Yes, DNS traffic is restricted: Resolving internet hosts is not possible.

Awesome - the problem is likely in that error path then.
Comment 14 Patrick McManus [:mcmanus] 2012-10-16 10:14:34 PDT
If I break resolution on my local computer I can reproduce the problem, and verify a fix. will69, michael-mozilla - thank you so much for the logs; they were key to finding the problem.

Test nightlies are available here if you wish to confirm the fix.

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.com-b30093f4dfd3/try-win32/
Comment 15 Patrick McManus [:mcmanus] 2012-10-16 10:18:19 PDT
Created attachment 671915 [details] [diff] [review]
patch 0

Christian, normally I would put this in honza's queue - but he is out for the week and as we probably want to port this to beta (where the regression originates) before it hits release every day counts. Its a tiny change, hope you can help.

The offending code is part of this patch - 
https://hg.mozilla.org/mozilla-central/rev/959f9da9f85e

The impact of it is that a DNS failure became a hard error when it should have not stopped processing.
Comment 16 Jason Duell [:jduell] (needinfo? me) 2012-10-16 11:37:39 PDT
Comment on attachment 671915 [details] [diff] [review]
patch 0

Review of attachment 671915 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.
Comment 17 Patrick McManus [:mcmanus] 2012-10-16 12:20:29 PDT
https://hg.mozilla.org/integration/mozilla-inbound/rev/073b904aed0c
Comment 18 Patrick McManus [:mcmanus] 2012-10-16 12:39:44 PDT
Comment on attachment 671915 [details] [diff] [review]
patch 0

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 767158 (introduced in ff17)
User impact if declined: Some users won't be able to browse the web at all with firefox. impacted users would be using a proxy/firewall that requires windows integrated authorization (aka kerberos, aka sspi, sometimes aka "negotiate") and prevents internet DNS inside the firewall
Testing completed (on m-c, etc.): problem reproduced (and fixed) and correlated with 2 different reporter logs.
Risk to taking this patch (and alternatives if risky): low.. 
String or UUID changes made by this patch:  none
Comment 19 PaulD 2012-10-16 13:13:03 PDT
hello,
this patch will he integrated in FF17? because I have the same problem on bluecoat proxies with authentication negotiate / kerberos / ntlm.
The problem is solved in FF19 nightly (19.0a1 (2012-10-16))
Thanks.
Comment 20 Patrick McManus [:mcmanus] 2012-10-16 13:50:21 PDT
(In reply to PaulD from comment #19)
> hello,
> this patch will he integrated in FF17? because I have the same problem on
> bluecoat proxies with authentication negotiate / kerberos / ntlm.
> The problem is solved in FF19 nightly (19.0a1 (2012-10-16))
> Thanks.

PaulD, Thanks for the info. Backporting is what the approval? flags on the patch are for - I have nominated the patch for porting to 18 and 17. The release team regularly triages all patches with those nominations and decides whether or not they should be backported.
Comment 21 michael-mozilla 2012-10-16 14:16:32 PDT
Hi Patrick,

Can confirm fixed with your nightly build.
Comment 22 will69 2012-10-17 00:39:46 PDT
Tryserver build fixes the problem. Thanks a lot Patrick.
Comment 23 :Ehsan Akhgari (busy, don't ask for review please) 2012-10-17 11:59:07 PDT
https://hg.mozilla.org/mozilla-central/rev/073b904aed0c
Comment 24 Ryan VanderMeulen [:RyanVM] 2012-10-17 12:13:17 PDT
Should this have a test?
Comment 25 Alex Keybl [:akeybl] 2012-10-17 16:35:47 PDT
Comment on attachment 671915 [details] [diff] [review]
patch 0

[Triage Comment]
Approving for Aurora/Beta because this is a low-risk fix to a new regression. If we see any fallout from this, we should strongly consider just backing out bug 767158. Please land as soon as possible.
Comment 26 Patrick McManus [:mcmanus] 2012-10-18 05:54:23 PDT
  https://hg.mozilla.org/releases/mozilla-beta/rev/36eb48b20b27
Comment 27 Patrick McManus [:mcmanus] 2012-10-18 05:59:29 PDT
https://hg.mozilla.org/releases/mozilla-aurora/rev/9364fc859c40
Comment 28 Tom 2012-10-25 01:43:07 PDT
*** Bug 804574 has been marked as a duplicate of this bug. ***
Comment 29 Patrick McManus [:mcmanus] 2012-10-29 12:11:18 PDT
backout of 767158 and 785050 from ff17 beta
  https://hg.mozilla.org/releases/mozilla-beta/rev/757f408c1494
Comment 30 Patrick McManus [:mcmanus] 2012-10-30 05:17:20 PDT
I'm not sure why scoobidiver changed this from fixed to affected for ff17, but since it is tracked for ff17 I will be clear:

785050 was a in place fix for a regression from 767158.

There was nothing wrong with 785050, but due to further problems stemming from 767158, 767158 and 785050 were backed out for ff17 beta.

Therefore there was a problem and that problem is still fixed (i.e. no regression).
Comment 31 Ryan VanderMeulen [:RyanVM] 2012-10-30 09:07:03 PDT
Setting status-firefox17 back to fixed based on comment 30.
Comment 32 Patrick McManus [:mcmanus] 2012-12-03 13:11:07 PST
bug 8040605 requires backing out 767158 from beta 18.. this bug is a spotfix to 767158 so it should come out of 18 too, but as the root problem is backed out too 18 changes to unaffected.
Comment 33 Scoobidiver (away) 2013-01-22 10:07:44 PST
Was it backed out from Beta and Aurora to fix bug 804605?
Comment 34 Patrick McManus [:mcmanus] 2013-01-22 10:10:27 PST
(In reply to Scoobidiver from comment #33)
> Was it backed out from Beta and Aurora to fix bug 804605?

It was, but you don't need to reopen this bug as a separate item -it was just a follow fix and when the code goes back into the tree it will all go in together. (its off all branches right now.)

Note You need to log in before you can comment on or make changes to this bug.