791008 - severe (20 second) jank with NTLM on osx/linux with gssapi

Status: RESOLVED DUPLICATE of bug 890908

(Core :: Networking: HTTP, defect, P3)

Description

[Patrick McManus [:mcmanus]]

linux (confirmed) and (I believe) os x use nsAuthGSSAPI.cpp to do windows auth such as NTLM. Windows uses different routines.

The first time I tried to diagnose a problem with this code, I saw a 20 second chrome hang from gss_init_sec_context while that routine tried to do a reverse dns lookup (directly using the system libraries) that timed out.

On the main thread that kills everything. Even off the main thread it wouldn't work very well. But even with names with all the right DNS entries this means main thread blocking network access is going on under the covers.

The saving grace here is probably windows auth is only lightly used in non windows environments.

Comment 1

[Patrick McManus [:mcmanus]]

#0  0x00007ffff70e69bd in read () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff707b0f8 in _IO_new_file_underflow (fp=0x7fffceea9c00) at fileops.c:619
#2  0x00007ffff707c13e in _IO_default_uflow (fp=0x7fffceea9c00) at genops.c:440
#3  0x00007ffff707028a in _IO_getline_info (fp=0x7fffceea9c00, buf=0x7fffffff7bc0 "\360}\377\377\377\177", n=255, delim=10, extract_delim=1, eof=0x0) at iogetline.c:74
#4  0x00007ffff706f16b in _IO_fgets (buf=0x7fffffff7bc0 "\360}\377\377\377\177", n=<optimized out>, fp=0x7fffceea9c00) at iofgets.c:58
#5  0x00007fffcb8ab11f in ?? () from /lib/libnss_mdns4.so.2
#6  0x00007fffcb8ab84f in _nss_mdns4_gethostbyaddr_r () from /lib/libnss_mdns4.so.2
#7  0x00007ffff710c972 in __gethostbyaddr_r (addr=0x7fffceea33b4, len=4, type=2, resbuf=0x7fffffff8530, buffer=0x7fffffff7f30 "\377\002", buflen=1024, result=0x7fffffff8570, h_errnop=0x7fffffff8590) at ../nss/getXXbyYY_r.c:256
#8  0x00007ffff7112e07 in __GI_getnameinfo (sa=0x7fffceea33b0, addrlen=<optimized out>, host=0x7fffffff8650 " \321\354\367\377\177", hostlen=1025, serv=0x0, servlen=0, flags=8) at getnameinfo.c:223
#9  0x00007fffcc7689eb in krb5_sname_to_principal () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#10 0x00007fffcc9de2d5 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#11 0x00007fffcc9d08b1 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#12 0x00007fffcc9d19ee in gss_init_sec_context () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#13 0x00007fffcc9ede95 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#14 0x00007fffcc9f082a in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#15 0x00007fffcc9d1a8d in gss_init_sec_context () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#16 0x00007ffff3c362e7 in nsAuthGSSAPI::GetNextToken (this=0x7fffcecfabe0, inToken=0x0, inTokenLen=0, outToken=0x7fffffffb198, outTokenLen=0x7fffffffb1bc) at ../../../scratch/extensions/auth/nsAuthGSSAPI.cpp:451
#17 0x00007ffff3c34bd1 in nsHttpNegotiateAuth::GenerateCredentials (this=0x7fffceea4a00, authChannel=0x7fffd0373310, challenge=0x7fffffffb580 "Negotiate", isProxyAuth=true, domain=0x0, username=0x0, password=0x0, sessionState=0x7fffffffb248, 
    continuationState=0x7fffd0375560, flags=0x7fffffffb264, creds=0x7fffffffb328) at ../../../scratch/extensions/auth/nsHttpNegotiateAuth.cpp:252
#18 0x00007ffff2884a40 in nsHttpChannelAuthProvider::GenCredsAndSetEntry (this=0x7fffd0375500, auth=0x7fffceea4a00, proxyAuth=true, scheme=0x7fffffffb460 "http", host=0x7fffceea4e68 "192.168.16.239", port=8080, directory=0x7fffffffb400 "", 
    realm=0x7fffffffb3a0 "", challenge=0x7fffffffb580 "Negotiate", ident=..., sessionState=..., result=0x7fffffffb328) at ../../../../scratch/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp:359
#19 0x00007ffff28861bb in nsHttpChannelAuthProvider::GetCredentialsForChallenge (this=0x7fffd0375500, challenge=0x7fffffffb580 "Negotiate", authType=0x7fffceea4928 "negotiate", proxyAuth=true, auth=0x7fffceea4a00, creds=...)
    at ../../../../scratch/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp:805
#20 0x00007ffff28855ac in nsHttpChannelAuthProvider::GetCredentials (this=0x7fffd0375500, challenges=0x7fffcee7e648 "Negotiate\nKerberos\nNTLM", proxyAuth=true, creds=...) at ../../../../scratch/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp:560
#21 0x00007ffff2883eeb in nsHttpChannelAuthProvider::ProcessAuthentication (this=0x7fffd0375500, httpStatus=407, SSLConnectFailed=false) at ../../../../scratch/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp:138
#22 0x00007ffff28683fb in mozilla::net::nsHttpChannel::ProcessResponse (this=0x7fffd0373000) at ../../../../scratch/netwerk/protocol/http/nsHttpChannel.cpp:1276
#23 0x00007ffff2877015 in mozilla::net::nsHttpChannel::OnStartRequest (this=0x7fffd0373000, request=0x7fffd4f50c80, ctxt=0x0) at ../../../../scratch/netwerk/protocol/http/nsHttpChannel.cpp:4807
#24 0x00007ffff274b608 in nsInputStreamPump::OnStateStart (this=0x7fffd4f50c80) at ../../../../scratch/netwerk/base/src/nsInputStreamPump.cpp:417
#25 0x00007ffff274b3bf in nsInputStreamPump::OnInputStreamReady (this=0x7fffd4f50c80, stream=0x7fffced1e6f8) at ../../../../scratch/netwerk/base/src/nsInputStreamPump.cpp:368
#26 0x00007ffff4182d05 in nsInputStreamReadyEvent::Run (this=0x7fffceea3500) at ../../../scratch/xpcom/io/nsStreamUtils.cpp:82
#27 0x00007ffff41a5338 in nsThread::ProcessNextEvent (this=0x7ffff6c6f160, mayWait=false, result=0x7fffffffb9ff) at ../../../scratch/xpcom/threads/nsThread.cpp:612
#28 0x00007ffff4137158 in NS_ProcessNextEvent_P (thread=0x7ffff6c6f160, mayWait=false) at nsThreadUtils.cpp:220
#29 0x00007ffff3ec96ba in mozilla::ipc::MessagePump::Run (this=0x7fffe8081300, aDelegate=0x7ffff6cc4120) at ../../../scratch/ipc/glue/MessagePump.cpp:82
#30 0x00007ffff41f7aa3 in MessageLoop::RunInternal (this=0x7ffff6cc4120) at ../../../scratch/ipc/chromium/src/base/message_loop.cc:215
#31 0x00007ffff41f7a34 in MessageLoop::RunHandler (this=0x7ffff6cc4120) at ../../../scratch/ipc/chromium/src/base/message_loop.cc:208
#32 0x00007ffff41f7a0d in MessageLoop::Run (this=0x7ffff6cc4120) at ../../../scratch/ipc/chromium/src/base/message_loop.cc:182
#33 0x00007ffff3d4c062 in nsBaseAppShell::Run (this=0x7fffe336e390) at ../../../scratch/widget/xpwidgets/nsBaseAppShell.cpp:163
#34 0x00007ffff3a7fdd6 in nsAppStartup::Run (this=0x7fffe3348880) at ../../../../scratch/toolkit/components/startup/nsAppStartup.cpp:290
#35 0x00007ffff26f0725 in XREMain::XRE_mainRun (this=0x7fffffffbeb0) at ../../../scratch/toolkit/xre/nsAppRunner.cpp:3792
#36 0x00007ffff26f0a01 in XREMain::XRE_main (this=0x7fffffffbeb0, argc=4, argv=0x7fffffffe318, aAppData=0x437c20) at ../../../scratch/toolkit/xre/nsAppRunner.cpp:3858
#37 0x00007ffff26f0c42 in XRE_main (argc=4, argv=0x7fffffffe318, aAppData=0x437c20, aFlags=0) at ../../../scratch/toolkit/xre/nsAppRunner.cpp:3933
#38 0x0000000000402a8f in do_main (argc=4, argv=0x7fffffffe318) at ../../../scratch/browser/app/nsBrowserApp.cpp:174
#39 0x0000000000402d45 in main (argc=4, argv=0x7fffffffe318) at ../../../scratch/browser/app/nsBrowserApp.cpp:279

Comment 2

[Patrick McManus [:mcmanus]]

andrew - is this something you might be able to help with?

Comment 3

[Andrew Bartlett]

Sorry for the very, very long delay looking into this. 

I certainly can take a look.  In general, my view is as I suggested in bug 1108971, that neither mozilla, nor the GSSAPI libs should not trying a reverse DNS lookup when in a Keberos environment.  

I realise however it has become standard practice in non-AD environments.  The workaround in the reporters case may be a simple as setting 
[libdefaults]
rdns = false

in the krb5.conf.

However, we may be able to invoke the Kerberos libs in a way that also suggests not to do DNS lookups, and I'll investigate that.

Comment 4

[Honza Bambas (:mayhemer)]

Note these days we have https://dxr.mozilla.org/mozilla-central/rev/3f4c3a3cabaf94958834d3a8935adfb4a887942d/netwerk/protocol/http/nsIHttpAuthenticator.idl#

implemented only for Negotiate in bug 890908.

Comment 5

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 6

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 7

[Honza Bambas (:mayhemer)]

More carefully inspecting the stack in comment 0 clearly tells me that this is duplicate of bug 890908.

There days nsHttpChannelAuthProvider::GenCredsAndSetEntry calls nsHttpNegotiateAuth::GenerateCredentialsAsync and all of this is processed on a worker thread.