1685427 - Crash when authenticating using SPNEGO on macOS (ARM)

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[Tomáš Kovár]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

1. Firefox 84.0.2 on macOS 11.1, running in ARM mode

2. Configured parameters for Kerberos SPENGO:

network.negotiate-auth.trusted-uris: https://

3. Open a webpage with SPNEGO authentication. I'm using Keycloak as the authentication server, federated with AD domain, with SPNEGO enabled.

Additional observations:

Under Rosetta, it works correctly. The crash only happens in ARM mode.

Actual results:

When opening the Keycloak auth page, the browser crashes. After the crash, Kerberos Credentials Cache (kcc) does not contain the appropriate ticket.

Expected results:

Browser should authenticate using SPNEGO, without any crash.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Haik Aftandilian [:haik]]

@Tomáš, thanks for the report. Does this log a crash in about:crashes? If so, could you send the link to the crash report?

Comment 3

[Tomáš Kovár]

@Haik, yes, it does. It is crash report bp-a86ad289-d545-47df-ad2b-8f8720210106.

Comment 5

[Kershaw Chang [:kershaw]]

Jens, could you find someone to take look at this?
I'm afraid there is no one in necko understands this code for now.

Comment 6

[Haik Aftandilian [:haik]]

I took a look at this and it appears to be a problem with the gss_indicate_mechs() function from the GSS library loaded from /System/Library/Frameworks/GSS.framework/Versions/A/GSS. The GSS API's appear to use opaque types which we've redefined in our tree and use to dereference and check internal values. I want to do a bit more verification and then I'll file an issue with Apple if necessary.

We're crashing in memcmp() while dereferencing the x0 register which holds or indexes into the first argument to memcmp() here https://searchfox.org/mozilla-central/rev/519f913527b0d9d5097d290d5731cff6b2991fe0/extensions/auth/nsAuthGSSAPI.cpp#295 so it's the item->elements pointer argument that is causing the crash.

Stepping through the debugger:

(lldb) n
Process 6232 stopped
* thread #1, name = 'MainThread', queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000102558578 XUL`nsAuthGSSAPI::nsAuthGSSAPI(this=0x000000013f921d20, package=PACKAGE_TYPE_NEGOTIATE) at nsAuthGSSAPI.cpp:292:31 [opt]
   289 	  if (GSS_ERROR(majstat)) return;
   290 	
   291 	  if (mech_set) {
-> 292 	    for (i = 0; i < mech_set->count; i++) {
   293 	      item = &mech_set->elements[i];
   294 	      if (item->length == gss_spnego_mech_oid_desc.length &&
   295 	          !memcmp(item->elements, gss_spnego_mech_oid_desc.elements,

Dumping out the mech_set contents shows that the item elements (within the mech_set->elements buffer) contain invalid pointers for some of the entries. These values appear to come from the gss_indicate_mechs() in the GSS library.

(lldb) expr -T -- *mech_set
(gss_OID_set_desc_struct) $2 = {
  (size_t) count = 7
  (gss_OID) elements = 0x000000015238e1d0
}

(lldb) expr -T -- (*mech_set).elements[0]
(gss_OID_desc_struct) $4 = {
  (OM_uint32) length = 9
  (void *) elements = 0x8eb89b5300000000
}

(lldb) expr -T -- (*mech_set).elements[1]
(gss_OID_desc_struct) $5 = {
  (OM_uint32) length = 1
  (void *) elements = 0x0000000000000006   <------------------
}

(lldb) expr -T -- (*mech_set).elements[2]
(gss_OID_desc_struct) $6 = {
  (OM_uint32) length = 2394471506
  (void *) elements = 0x0000000600000001    <------------------
}

Comment 7

[Haik Aftandilian [:haik]]

I can't reproduce this with a simple test program that calls gss_indicate_mechs(). And I tried building Firefox without jemalloc enabled to see if that could be triggering the problem, but that did not affect the crash. I can't rule out a Firefox issue causing this, but I've filed Apple feedback request FB8969439 and let our Apple contact know about the problem.

Comment 8

[Haik Aftandilian [:haik]]

Apple has looked into this and they don't see anything on their side that would explain the invalid pointers. So this may be more likely to be a Firefox or configuration issue.

@Tomáš, could you provide us with the output from /System/Library/PrivateFrameworks/Heimdal.framework/Helpers/gsstool supported-mechanisms and also the contents of /etc/gss/mech if the file is present?

Comment 9

[Haik Aftandilian [:haik]]

After some more discussion with Apple, the problem appears to be related to the use of #pragma pack. Specifically, the Apple gssapi.h header file at

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/gssapi/gssapi.h

does not apply the 2-byte packing to arm64 builds:

    70 #if defined(__APPLE__) && (defined(__ppc__) || defined(__ppc64__) || defined(__i386__) || defined(__x86_64__))
    71 #    pragma pack(push,2)                                                         
    72 #endif

However, in Firefox[1], we apply it to all macOS builds regardless of the architecture.

We're waiting to hear if omitting arm64 in Apple's gssapi.h is intentional or not.

1. https://searchfox.org/mozilla-central/rev/03224522336f60a1a61a87e1fcd4feb0a0315a9b/extensions/auth/gssapi.h#97-99

Comment 10

[Tomáš Kovár]

@Haik,

the output of gsstool supported-mechanisms is:

Supported mechanisms:
OID                     Name    SASL
1 2 840 113554 1 2 2    krb5    GS2-KRB5
1 3 6 1 5 5 2           SPNEGO  SPNEGO
1 2 752 43 14 2                 
1 3 6 1 5 5 14                  
1 3 6 1 4 1 311 2 2 10  NTLM    
1 3 5 1 5 2 7                   
1 3 6 1 5 2 5           iakerb  

The file /etc/gss/mech doesn't exists.

Another bit: Safari, ssh and Finder (for authenticating smb:// shares) all work.

Comment 11

[Haik Aftandilian [:haik]]

Thanks. I think we have a good understanding of the problem now. See comment 9 for details.

If you are comfortable testing a potential fix, you can download this developer build from our automated build infrastructure here.
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LLlyKuu_SumuB45FvgzLew/runs/0/artifacts/public/build/target.dmg

This build is not signed with our official release certificates so you will have to take an extra step to run it. Specifically, clear the quarantine flags by running xattr -cr Firefox\ Nightly.app/ from the terminal after extracting from the dmg.

You can also get to this download by visiting https://treeherder.mozilla.org/jobs?repo=try&revision=d2454136e506050fbdb4b52ccc22acca76743886 and clicking on the "OS X Cross Compiled shippable opt" Rpk build. Then clicking on artifacts, then target.dmg. And from that page you can also see the changes in the build by clicking on the cff919da3386 changeset link taking you to https://hg.mozilla.org/try/rev/cff919da338677801b3a018876b31f2d0b5770f4

Comment 12

[Haik Aftandilian [:haik]]

Apple acknowledged the header file issue. It's not clear if a future macOS version will change the header file to apply the packing to ARM64 as well. I suspect this will be left as is because changing it would cause binary incompatibility with ARM64 applications built using older macOS versions. I'm going to move forward with the fix to remove the #pragma pack for ARM64 builds.

Comment 13

[Haik Aftandilian [:haik]]

Attachment: Bug 1685427 - Crash when authenticating using SPNEGO on macOS (ARM) r?spohl!

Don't pragma pack gssapi structs on ARM64 to be consistent with macOS system gssapi.h header file.

Issue reported to reported to Apple and acknowledged.

Comment 14

[Haik Aftandilian [:haik]]

I'll look into whether or not we still need to use gssapi header files from the tree. If we used the system gssapi header files for Mac builds, we would not have run into this problem.

Comment 15

[Tomáš Kovár]

I've tried the nightly build above.

The good news is, that Firefox doesn't crash.

The bad news is, that it doesn't authenticate. After checking the credentials cache, there's no ticket for the http server. What's interesting, it doesn't authenticate in Rosetta mode either.

Comment 16

[Tomáš Kovár]

Sorry, my bad, error in network.negotiate-auth.trusted-uris. It does authenticate.

Thank you.

Comment 17

[Haik Aftandilian [:haik]]

(In reply to Tomáš Kovár from comment #16)

Sorry, my bad, error in network.negotiate-auth.trusted-uris. It does authenticate.

Thank you.

Awesome, thanks for checking. We should have this fix in Nightly within a few days and then will get this uplifted to Beta/Release when possible.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4f9faa827c24
Crash when authenticating using SPNEGO on macOS (ARM) r=spohl

Comment 19

[Haik Aftandilian [:haik]]

Comment on attachment 9198054 [details]
Bug 1685427 - Crash when authenticating using SPNEGO on macOS (ARM) r?spohl!

Beta/Release Uplift Approval Request

• User impact if declined: Users on new Apple Silicon devices with the browser configured to use SPNEGO authentication will experience browser crashes.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: It would be ideal if QE could verify this beyond the verification already done, but setting up an environment that uses SPNEGO authentication might be difficult. Given the low-risk nature of the fix and the verification already done, it is not a requirement. The reporter verified the fix with a developer build. I verified the fix using an instrumented build of Nightly that exercised the failing code.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It only affects Firefox when run on the new Apple Silicon devices and only when the browser has been configured to use SPNEGO authentication which requires modifying settings in about:config.
• String changes made/needed:

Comment 20

[Haik Aftandilian [:haik]]

Needinfo for comment 14.

Comment 21

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4f9faa827c24

Comment 22

[Julien Cristau [:jcristau] (back August 18)]

Comment on attachment 9198054 [details]
Bug 1685427 - Crash when authenticating using SPNEGO on macOS (ARM) r?spohl!

moving uplift request to release in case of a RC respin or dot release.

Comment 24

[Alin Ilea, Desktop Test Engineering [pto until 01.09]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0

Hi,

I tried to reproduce the issue in order to confirm the fix, but given the difficult environment configurations I did not manage to reproduce the issue and confirm the fix. Based on comment 15 and comment 19, the issue is no longer reproducible in the reporter's environment.

Thanks.

Comment 26

[Julien Cristau [:jcristau] (back August 18)]

Comment on attachment 9198054 [details]
Bug 1685427 - Crash when authenticating using SPNEGO on macOS (ARM) r?spohl!

approved for 85.0.1

Comment 27

[Julien Cristau [:jcristau] (back August 18)]

https://hg.mozilla.org/releases/mozilla-release/rev/e2cb6d1f4431

Comment 28

[Julien Cristau [:jcristau] (back August 18)]

Adding to the 85.0.1 release notes:
"Fixed a crash when authenticating to websites using SPNEGO on macOS devices with Apple Silicon CPUs (bug 1685427)."

Comment 29

[Haik Aftandilian [:haik]]

Filed bug 1693136 for possible header file improvements.