This is a bug in extensions/auth/nsHttpNegotiateAuth.cpp When used against a site with a round-robin DNS, and a GSSAPI library which canonicalises hostnames (such as MIT Kerberos), the HTTP-Negotiate code will fail to work. Take a server SERVICE, which is a cname for HOST-A and HOST-B. When the HTTP library connects to SERVICE, it does a DNS lookup, gets HOST-A returned, and so opens it's connection to HOST-A. When that service then requests Negotiate authentication, the Negotiate library is started and asks the GSSAPI layer to create a security context with SERVICE. If the GSSAPI layer uses canonicalisation, it then asks the DNS to canonicalise SERVICE, and gets HOST-B back as the hostname to use, due to the round-robin. So, you end up with a connection to HOST-A, but using HOST-B's GSSAPI identity. Needless to say, this rarely works. The fix for this is to know whether the GSSAPI is going to canonicalise or not - which is probably yet-another-preference, as there's no way of telling this externally. Then, when we are using a canonicalising GSSAPI library, use the name of the current connection endpoint, rather than the service name from the uri when calling into GSSAPI. Is it possible to get the name of the endpoint out of the current nsIHttpChannel API?
IE appears to suffer from this affliction also and MS have released a hotfix for it http://support.microsoft.com/kb/911149 which I found from here http://blogs.msdn.com/saurabh_singh/archive/2007/01/29/kerberos-troubleshooting-from-iis-perspective.aspx
SSPI does not canonicalize the hostname for the exact reason you have described. I'd advise you to turn that off too for MIT Kerberos and it will work. Set in krb5.conf rdns to false.