Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() with Mac Sierra. Need to disable ntlm_auth

VERIFIED FIXED in Firefox 56

Status

()

Core
Networking
--
critical
VERIFIED FIXED
a year ago
4 months ago

People

(Reporter: jesup, Assigned: bagder)

Tracking

({crash, regression, topcrash-thunderbird})

53 Branch
mozilla57
Unspecified
Mac OS X
crash, regression, topcrash-thunderbird
Points:
---

Firefox Tracking Flags

(thunderbird_esr5256+ fixed, firefox-esr52 wontfix, firefox53+ wontfix, firefox54+ wontfix, firefox55+ wontfix, firefox56+ fixed, firefox57+ fixed)

Details

(Whiteboard: [ntlm][necko-backlog][startupcrash][tbird topcrash], crash signature)

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

a year ago
This bug was filed from the Socorro interface and is 
report bp-48488ba5-c600-444a-9cdf-bf92a0170425.
=============================================================

This is for the subset of the signature where nsAuthSambaNTLM::SpawnNTLMAuthHelper() (in extensions/) was on the stack.  Last call outside the kernel was to fork() in _MD_CreateUnixProcess().

Lots of crashes in the last week - ~450 -- all in 52 and newer, including a bunch in Thunderbird 52.
(Reporter)

Comment 1

a year ago
No idea who should get this, but the crash has spiked and is recent
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(jduell.mcbugs)
This is spiking on 52.0.2 _and_ on 45.8.0 (~800 crashes in 6 months).  Probably one of the [NTLM] fixes landed on or around 52 causes this.
Assignee: nobody → honzab.moz
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(jduell.mcbugs)
Summary: Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() → [ntlm] Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper()
Summary: [ntlm] Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() → Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper()
Whiteboard: [ntlm]
Whiteboard: [ntlm] → [ntlm][necko-active]

Updated

a year ago
Duplicate of this bug: 1361966

Comment 4

a year ago
[Tracking Requested - why for this release]:
this issue is a sizeable crash on 53.0 where it's around 10% of browser crashes for users on macos.

the issue seems to affect only users on mac os 10.12. it also looks like a regression in firefox 53 (as there may have been a regeneration of crash signatures since the bug was created - the signature of the report in comment #0 apparently changed), crashes with version 52 seem to all come from thunderbird:
https://crash-stats.mozilla.com/search/?proto_signature=~nsAuthSambaNTLM%3A%3ASpawnNTLMAuthHelper&product=Firefox&platform=Mac%20OS%20X&date=%3E%3D2016-11-04T10%3A54%3A00.000Z&_facets=signature&_facets=version&_facets=user_comments&_facets=install_time&_facets=platform_version#facet-version

the issue triggers a lot of user comments indicating that it is a severe (& probably reproducible) issue for affected users, many of them tying the crash to being connected through a proxy/vpn/corporate firewall:
https://crash-stats.mozilla.com/signature/?product=Firefox&proto_signature=~nsAuthSambaNTLM%3A%3ASpawnNTLMAuthHelper&signature=__pthread_kill&date=%3E%3D2017-03-01T00%3A00%3A00.000Z#comments
Crash Signature: [@ libsystem_kernel.dylib@0x19d42] → [@ libsystem_kernel.dylib@0x19d42] [@ __pthread_kill] [@ libsystem_kernel.dylib@0x19dd6] [@ libsystem_kernel.dylib@0x19dda]
status-firefox53: --- → affected
status-firefox54: --- → affected
status-firefox55: --- → ?
status-firefox-esr52: --- → unaffected
tracking-firefox53: --- → ?
tracking-firefox54: --- → ?
Keywords: regression
Version: unspecified → 53 Branch
Makes sense to track this for 54/55, for investigation. Not sure we would be able to do much about 53.

What are the next steps here?
status-firefox55: ? → affected
tracking-firefox54: ? → +
tracking-firefox55: --- → +
This looks pretty bad for Mac users. This seems to be the most common signature, for MacOS 10.12 https://crash-stats.mozilla.com/signature/?signature=__pthread_kill

If we fix this it could be a ridealong on 53 if we end up with another desktop dot release. No plans for one yet, though.
tracking-firefox53: ? → +

Updated

a year ago
Crash Signature: [@ libsystem_kernel.dylib@0x19d42] [@ __pthread_kill] [@ libsystem_kernel.dylib@0x19dd6] [@ libsystem_kernel.dylib@0x19dda] → [@ libsystem_kernel.dylib@0x19d42] [@ __pthread_kill] [@ libsystem_kernel.dylib@0x19dd6] [@ libsystem_kernel.dylib@0x19dda] [@ __pthread_kill | abort | free | _arc4_fork_child ]
Hi Selena, I pinged you on IRC about this top crasher. It came up in the channel meeting today. We would like some dev help here to investigate and potentially fix this top crasher. Thanks!
Flags: needinfo?(sdeckelmann)
It would be great if someone could try to reproduce this.  I don't currently own any OSX running machine and don't have an easy way to install a virtual.

I can *try* to provide a public test page to trigger this issue (no promises it would be reliable to reproduce, tho).  Other way is to setup IIS server on a windows machine (7-10 Pro) and setup Windows Integrated authentication on some folder exposed via HTTP.  It's relatively easy, no need for domain controllers at all.
Assignee: honzab.moz → nobody
Whiteboard: [ntlm][necko-active] → [ntlm]
So, the page you may try to reproduce the crash with is at:

https://janbambas.cz/moz/ntlm/
So :mhowell just tried to reproduce with both a successful and unsuccessful login in win10 Pro and could not. I sent email to pi-request for additional help.
To clarify, I was running IIS on Win10 Pro and logging into it from a macOS 10.12 system.
(In reply to Honza Bambas (:mayhemer) from comment #9)
> So, the page you may try to reproduce the crash with is at:
> 
> https://janbambas.cz/moz/ntlm/

What's needed to reproduce? I tried this site both with "cancel" and entering blank or dummy creds and didnt see a crash. Im on OSX 10.12.24 with Firefox 53 (release) and nightly (buildid: 20170516122050)
Whiteboard: [ntlm] → [ntlm][necko-backlog]
(In reply to Paul Theriault [:pauljt] from comment #12)
> (In reply to Honza Bambas (:mayhemer) from comment #9)
> > So, the page you may try to reproduce the crash with is at:
> > 
> > https://janbambas.cz/moz/ntlm/
> 
> What's needed to reproduce? I tried this site both with "cancel" and
> entering blank or dummy creds and didnt see a crash. Im on OSX 10.12.24 with
> Firefox 53 (release) and nightly (buildid: 20170516122050)

Thanks for looking into this.

Try adding "https://janbambas.cz" to "network.automatic-ntlm-auth.trusted-uris" pref.  Looking into the code, that's the necessary condition to trigger the crashing code path.
Flags: needinfo?(ptheriault)
I made that change, and got a system but not a firefox crash. I'll pass it along out of band.
Sorry Honza, I first assumed that Selena had given you all you needed, but maybe that wasnt the case. ANyways, I just tested, setting the pref as you said in comment 13 and I'm not able to reproduce on OSX (nightly)

I tried both https://janbambas.cz/moz/ntlm/ and https://janbambas.cz/moz/ntlm/?non-sticky, both with and without creds, and also cancelling the prompt for creds.
Flags: needinfo?(ptheriault)
(In reply to Paul Theriault [:pauljt] from comment #15)
> Sorry Honza, I first assumed that Selena had given you all you needed, but
> maybe that wasnt the case. ANyways, I just tested, setting the pref as you
> said in comment 13 and I'm not able to reproduce on OSX (nightly)
> 
> I tried both https://janbambas.cz/moz/ntlm/ and
> https://janbambas.cz/moz/ntlm/?non-sticky, both with and without creds, and
> also cancelling the prompt for creds.

She has sent me a "system crash" with not more data to gain from than from regular crash reporter records.  Also, OSX is not my development platform at all and this is an OS or near-OS problem, not a networking one as it appears. 

The first build id having this crash is 
Firefox 20170413192749, version 53.0. (m-r@d345b657d381)
Thunderbird 20170413214957, version 52.0.1

Hence, quite broad.

The range 52.0 Firefox to 53.0 Firefox is m-r@9a3506a37e59 - m-r@d345b657d381.  There is no change to NSPR in 53.0.  Only change I found close to this code between 52 and 53 is bug 1344305, but it's off the table, since we have a TB 52.0.1 crash based on esr52 _not_ containing that fix.

Hence, I really don't know what more to do.  Someone needs to debug this and see if the the handle we are forking is invalid and understand exactly what the crash is - is it a access violation or is it a system assertion failure or something else?  Also I don't understand the NSPR code being in fault here and can't say who would be the best person to debug that part of the code and give more clues :(
Too late at this point for 53. And probably for 54 unless we find someone else who digs into this and a way to test better.
status-firefox53: affected → wontfix
Jim -- is this something your team might be able to dig into?
Flags: needinfo?(sdeckelmann) → needinfo?(jmathies)
Assignee: nobody → daniel
Selena, just FYI, Daniel is willing to try to reproduce this bug and if successful, debug what's going on.
(Assignee)

Comment 20

a year ago
On my mac mini running the latest macOS (10.12.5), I tried the https://janbambas.cz/moz/ntlm/ link (after editing network.automatic-ntlm-auth.trusted-uris as comment #13 suggests).

I just tested with Firefox 53, Firefox Nightly 55 and my own local build straight off github/gecko-dev this morning.

Cannot reproduce! =(
You may ask Selena or maybe you can dig something out from her email I forwarded to you.
Tracy, can you try to repo this please? If not, ping spohl, see if he can. 

- running osx 10.12, fx 53
- add "https://janbambas.cz" to "network.automatic-ntlm-auth.trusted-uris" pref
- visit https://janbambas.cz/moz/ntlm/
Flags: needinfo?(twalker)
No crash here with Firefox 53.0.3 on macOSX 10.12.5.

I get Authorization: NTLM Type3 -> OK, you successfully authenticated via NTLM with a dummy password.
Flags: needinfo?(twalker)
(In reply to Tracy Walker [:tracy] from comment #23)
> No crash here with Firefox 53.0.3 on macOSX 10.12.5.
> 
> I get Authorization: NTLM Type3 -> OK, you successfully authenticated via
> NTLM with a dummy password.

Exactly the same environment/result here.

Updated

a year ago
Flags: needinfo?(jmathies)

Updated

11 months ago
See Also: → bug 1362238

Comment 25

11 months ago
Since 54 was released, mark 54 won't fix.
status-firefox54: affected → wontfix
(Assignee)

Comment 26

11 months ago
The above crash reports seem to never occur on versions >54 but still keeps happening on 54. (Or it changed signature enough to not look like the same anymore.)

Updated

11 months ago
Duplicate of this bug: 1379443

Comment 28

11 months ago
__pthread_kill | abort | free | _arc4_fork_child is #4 crash for Macs in TB52.2.1

Signature in 54.0b3 is libsystem_kernel.dylib@0x19d42  bp-434b2515-7a48-4195-92e3-a92d50170709
status-thunderbird_esr52: --- → affected
Keywords: topcrash-thunderbird
Whiteboard: [ntlm][necko-backlog] → [ntlm][necko-backlog][startupcrash][tbird topcrash]

Updated

11 months ago
Blocks: 1362238
See Also: bug 1362238
Summary: Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() → Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() with Mac Sierra

Comment 29

11 months ago
No crash on Firefox 54.0.1 in macOS 10.12.5.

Removing myself from the CC list. needinfo-me if another test is needed.

:)

Comment 30

10 months ago
Still crashing in Firefox 54.0.1 on macOS 10.12.5 (16F73).

I see 744 crashes (link below) matching the following:

  * "signature" is "__pthread_kill | abort | free | _arc4_fork_child"
  * "proto signature" contains: "SpawnNTLMAuthHelper"
  * "product" has terms: "Firefox"
  * "version" has terms: "54.0.1"

All of the crashes appear to have occurred on macOS 10.12.

https://crash-stats.mozilla.com/signature/?product=Firefox&proto_signature=~SpawnNTLMAuthHelper&version=54.0.1&signature=__pthread_kill%20%7C%20abort%20%7C%20free%20%7C%20_arc4_fork_child&date=%3E%3D2017-06-18T18%3A13%3A12.000Z&date=%3C2017-12-18T18%3A13%3A00.000Z

Comment 31

10 months ago
my organization uses an internal app that reliably reproduces this crash.  let me know if I can help.

Comment 32

10 months ago
hi tquan, if you can easily reproduce the crash on a particular website or with a certain sequence of inputs, it would be great if you could run the mozregression tool from http://mozilla.github.io/mozregression/install on the affected device - it will automatically download & run various firefox versions and ask you after each one if it contained the bug or not until it ends up at a particular patch/change in the code that triggered this problem.

you can start the tool with "mozregression --good 52" and then work your way through it until it provides you with a pushlog or regressing bug. that would be the crucial piece of information to move this bug forward. thank you!

http://mozilla.github.io/mozregression/documentation/usage.html

Comment 33

10 months ago
Here's how you can reproduce it yourself

https://bugzilla.mozilla.org/show_bug.cgi?id=1359624#c13 is not correct.  That's not the right syntax for the "network.automatic-ntlm-auth.trusted-uris" pref (or at least not the one that triggers the bug)

to trigger the bug:

1.  for "network.automatic-ntlm-auth.trusted-uris" pref set it to simply "janbambas.cz" (without the quotes)
2.  visit the previously created test site https://janbambas.cz/moz/ntlm/
3.  Crash reporter appears

our organization used "network.automatic-ntlm-auth.trusted-uris" pref of the above format and it stopped working in some recent Firefox release on MacOS 10.12.  if I get some cycles I will try mozregression later today to figure out where the bug was introduced.  the documentation for this parameter does say that a simple domain (not a URL) is allowed as a value for this preference.  https://developer.mozilla.org/en-US/docs/Mozilla/Integrated_authentication says 

site-list = "mydomain.com, https://myotherdomain.com"

I'm on Firefox 54.0.1, MacOS 10.12.6.

Comment 34

10 months ago
btw, you might have to restart the browser after changing network.automatic-ntlm-auth.trusted-uris pref.

Comment 35

10 months ago
(In reply to [:philipp] from comment #32)
> hi tquan, if you can easily reproduce the crash on a particular website or
> with a certain sequence of inputs, it would be great if you could run the
> mozregression tool from http://mozilla.github.io/mozregression/install on
> the affected device - it will automatically download & run various firefox
> versions and ask you after each one if it contained the bug or not until it
> ends up at a particular patch/change in the code that triggered this problem.
> 
> you can start the tool with "mozregression --good 52" and then work your way
> through it until it provides you with a pushlog or regressing bug. that
> would be the crucial piece of information to move this bug forward. thank
> you!
> 
> http://mozilla.github.io/mozregression/documentation/usage.html

philipp, I couldn't reproduce the bug using mozregression, even using the profile that I used to reproduce the bug in https://bugzilla.mozilla.org/show_bug.cgi?id=1359624#c33 .  I'm not sure why it doesn't reproduce in mozregression but does reproduce with the actual browser.  Can you verify that you're able to reproduce the bug that way?  At least it's a start. As mentioned, NTLM is used in corporate environments quite a bit so it would be good to nail this one down.

Comment 36

10 months ago
sorry, i don't have any system with macos available, so i cannot try to reproduce it myself...
Could please somebody with a deeper OSX development experience sit down and find out what this crash actually is?  I believe the real problem is in the ntlm binary and not firefox.  OSX update probably exposes a bug in the ntlm binary or there is an additional check the ntlm binary doesn't pass anymore or something like that...

Adding some samba folks.

Comment 38

10 months ago
is it possible that the ntlm binary isn't included with mozregression, or that mozregression cannot "catch" crashes that happen with that binary?

Comment 39

10 months ago
I doubt you will find ntlm_auth on a MacOS box, they ditched Samba years ago (GPLv3), so this whole thing should be disabled, the internal NTLM support will have to do, and there won't be any cached passwords anyway (mac AD login never used winbind).
(In reply to Andrew Bartlett from comment #39)
> I doubt you will find ntlm_auth on a MacOS box, they ditched Samba years ago
> (GPLv3), so this whole thing should be disabled, the internal NTLM support
> will have to do, and there won't be any cached passwords anyway (mac AD
> login never used winbind).

Thanks, that's a good point.  So, it looks like we are facing an OSX bug here after all.  No idea why they are crashing instead of reporting a runtime error.

Selena, do you have ntlm_auth binary on the system you experienced the crash in comment 14?  If not, I think we have to check its existence on the system (I don't want to break anyone who potentially installed it manually by completely disabling this feature.)

Thanks.
Flags: needinfo?(sdeckelmann)

Comment 41

10 months ago
(In reply to Honza Bambas (:mayhemer) from comment #40)
> (In reply to Andrew Bartlett from comment #39)
> > I doubt you will find ntlm_auth on a MacOS box, they ditched Samba years ago
> > (GPLv3), so this whole thing should be disabled, the internal NTLM support
> > will have to do, and there won't be any cached passwords anyway (mac AD
> > login never used winbind).
> 
> Thanks, that's a good point.  So, it looks like we are facing an OSX bug
> here after all.  No idea why they are crashing instead of reporting a
> runtime error.
> 
> Selena, do you have ntlm_auth binary on the system you experienced the crash
> in comment 14?  If not, I think we have to check its existence on the system
> (I don't want to break anyone who potentially installed it manually by
> completely disabling this feature.)
> 
> Thanks.

Yup, Apple removed Samba from MacOS in all versions 10.7 and higher: http://appleinsider.com/articles/11/03/23/inside_mac_os_x_10_7_lion_server_apple_replaces_samba_for_windows_networking_services.html

I'm on MacOS 10.12, can reproduce the crash, and do *not* have the ntlm_auth binary on the system.
I can easily reproduce this with the steps in comment #33. I'm on Firefox 54.0.1, MacOS 10.12.5.

I observed some weird things. First, I can only reproduce this when I lunch firefox with Launchpad, but never reproduce this when launching firefox by running ./firefox-bin with my shell. Second, when crash reporter shows up, my firefox is still working. It looks like that the crashed process is another one but not firefox. Not sure if it's the same situation as commentt #14 described.

BTW, I can't reproduce this using nightly with the same profile, neither lunching firefox by Launchpad nor shell.

Let me know if there is anything I can help. Thanks.
(In reply to Kershaw Chang [:kershaw] from comment #42)
> Let me know if there is anything I can help. Thanks.

Do you have the ntlm_auth binary on that system?
Flags: needinfo?(kechang)
(In reply to Honza Bambas (:mayhemer) from comment #43)
> (In reply to Kershaw Chang [:kershaw] from comment #42)
> > Let me know if there is anything I can help. Thanks.
> 
> Do you have the ntlm_auth binary on that system?

No, I can't find this file on my system.
Flags: needinfo?(kechang)

Comment 45

10 months ago
(In reply to Kershaw Chang [:kershaw] from comment #42)
> I can easily reproduce this with the steps in comment #33. I'm on Firefox
> 54.0.1, MacOS 10.12.5.
> 
> I observed some weird things. First, I can only reproduce this when I lunch
> firefox with Launchpad, but never reproduce this when launching firefox by
> running ./firefox-bin with my shell. Second, when crash reporter shows up,
> my firefox is still working. It looks like that the crashed process is
> another one but not firefox. Not sure if it's the same situation as commentt
> #14 described.
> 
> BTW, I can't reproduce this using nightly with the same profile, neither
> lunching firefox by Launchpad nor shell.
> 
> Let me know if there is anything I can help. Thanks.

All of the above are true for me as well and other folks in my organization who can reproduce the bug. None of us have the ntlm_auth binary.

It does appear that this functionality likely never worked on Macs anyway as mentioned in comment #39. Even if we had the ntlm_auth binary the Mac just doesn't have this functionality. If that is the case can we simply disable the usage of ntlm_auth on Macs as the author of that comment suggests?
(In reply to Honza Bambas (:mayhemer) from comment #40)

> Selena, do you have ntlm_auth binary on the system you experienced the crash
> in comment 14?  If not, I think we have to check its existence on the system
> (I don't want to break anyone who potentially installed it manually by
> completely disabling this feature.)


I don't have that binary on my system
Flags: needinfo?(sdeckelmann)
Too late for 55 but happy to take a fix for 56
status-firefox55: affected → wontfix
status-firefox56: --- → affected
status-firefox57: --- → affected
tracking-firefox56: --- → +
tracking-firefox57: --- → +

Comment 48

9 months ago
Daniel, can you work up a patch?

Being a startup crash makes this pretty bad.  Sample of half the comments
 Crashes every time I start Thunderbird since the upgrade :-( :-(
 This new update is UNUSABLE! Please, please test this crap BEFORE you publish it. Looks like I'm going to have to find a new Email Client. Nice work folks.
 When I upgraded to this version during starting this message shows. The thunderbird is started and when I click exit the window with thunderbird is opened and it works. But it hangs a lot. It seams to me that it starts two thunderbirds and the second now is crushing.
Aparece esta ventana de informe de fallo cada vez que se abre Thunderbird. Gracias
Flags: needinfo?(daniel)
Summary: Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() with Mac Sierra → Crash in libsystem_kernel.dylib@0x19d42 called from nsAuthSambaNTLM::SpawnNTLMAuthHelper() with Mac Sierra. Need to disable ntlm_auth
(Assignee)

Comment 49

9 months ago
I've never been able to reproduce, and the crash stats from way before in this bug were only up to Firefox 54. I presume because the signature changed? Comment #47 implies this still happens in 56 and 57?
Assignee: daniel → nobody
Flags: needinfo?(daniel)
(Assignee)

Comment 50

9 months ago
(oops unassigned by mistake, but its not a bad idea for someone who can reproduce to take this)
Assignee: nobody → daniel
Daniel, the plan is to disable the nsAuthSambaNTLM module on OSX.  You can either do it in nsHttpNTLMAuth::ChallengeReceived (preferred - we already have some Windows specific #ifdefs there) or conditionally remove it at [1] from the factory list.


[1] https://dxr.mozilla.org/mozilla-central/rev/56188620cce00b19700fbb8efaafea65e6ca8c61/extensions/auth/nsAuthFactory.cpp#213
(Assignee)

Comment 52

9 months ago
Created attachment 8901678 [details] [diff] [review]
0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX-r-.patch

Honza, you means simply like this?
Attachment #8901678 - Flags: review?(honzab.moz)
Comment on attachment 8901678 [details] [diff] [review]
0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX-r-.patch

Review of attachment 8901678 [details] [diff] [review]:
-----------------------------------------------------------------

let's try it.
Attachment #8901678 - Flags: review?(honzab.moz) → review+

Comment 54

9 months ago
BTW this still reproduces in Firefox 55, but instead of a crash the browser hangs/becomes unresponsive.

I'm happy to test the fix once a build that has it is available...I assume some upcoming nightly build will get it?

Comment 55

9 months ago
(In reply to Tony Quan from comment #54)
> BTW this still reproduces in Firefox 55, but instead of a crash the browser
> hangs/becomes unresponsive.
> 
> I'm happy to test the fix once a build that has it is available...I assume
> some upcoming nightly build will get it?

Yes, the day after you see messages that Daniel/bot has landed it on mozilla-centra.

Updated

9 months ago
tracking-thunderbird_esr52: --- → ?
(Assignee)

Updated

8 months ago
Keywords: checkin-needed
(Assignee)

Comment 56

8 months ago
But ok, let's do a try first.
Keywords: checkin-needed
(Assignee)

Comment 57

8 months ago
Created attachment 8906530 [details] [diff] [review]
v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch

Turned out to be a good idea...

Here's the try build for this version: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b24f4232cf0a3b32d41a094fbdb398754c35c42&selectedJob=130004632
Attachment #8901678 - Attachment is obsolete: true
Attachment #8906530 - Flags: review+
(Assignee)

Updated

8 months ago
Keywords: checkin-needed

Comment 58

8 months ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3b924184c333
Disable nsAuthSambaNTLM module on OSX. r=mayhemer
Keywords: checkin-needed

Comment 59

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/3b924184c333
Status: NEW → RESOLVED
Last Resolved: 8 months ago
status-firefox57: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57

Comment 60

8 months ago
Jorg, 
I could be wrong, but I don't foresee Firefox taking this on esr.
do you see a need for this to go through beta before us taking for 52.x?
Flags: needinfo?(jorgk)
"A need", hmm, I don't watch crash stats, I watch C-C treeherder ;-)
I'll take your advice on taking that one-line change.
Flags: needinfo?(jorgk)

Comment 62

8 months ago
I was able to confirm with Nightly that this bug is fixed (no further crash/hang on NTLM authentication on MacOS Sierra).  the issue still reproduces for me on Firefox 55 with the same profile.
Please request Beta/ESR52 approval on this when you get a chance.
status-firefox-esr52: unaffected → affected
Flags: needinfo?(daniel)
(Assignee)

Comment 64

8 months ago
Comment on attachment 8906530 [details] [diff] [review]
v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined:

- It is a crashing bug that will keep annoying users (on mac).

Fix Landed on Version:

- in nightly/57

Risk to taking this patch (and alternatives if risky): 

- I'd call it very low risk since it doesn't change any actual code, just disables/removes a part.

String or UUID changes made by this patch: 

- none

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.
Flags: needinfo?(daniel)
Attachment #8906530 - Flags: approval-mozilla-esr52?
Attachment #8906530 - Flags: approval-mozilla-beta?

Comment 65

8 months ago
(In reply to Daniel Stenberg [:bagder] from comment #64)
> Comment on attachment 8906530 [details] [diff] [review]
> v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch
> 
> [Approval Request Comment]
> If this is not a sec:{high,crit} bug, please state case for ESR
> consideration:
> User impact if declined:
> 
> - It is a crashing bug that will keep annoying users (on mac).

I know you don't mean in the sense of "merely annoying".  So to clarify the impact from the Thunderbird perspective, this is a topcrash for Mac (3rd only behind bug 1365600 and bug 1365136), made worse because >60% of users crash on startup which effectively makes the current version of Thunderbird totally unusable for some class of Mac users. (The crash volume has gone down some each of the last few months, but that's probably from users going back to a prior version or abandoning Thunderbird)
(Assignee)

Comment 66

8 months ago
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #65)

> > - It is a crashing bug that will keep annoying users (on mac).
> 
> I know you don't mean in the sense of "merely annoying".

I mean it in the exact sense I wrote it. It crashes Firefox, which annoys users. What else do you think I should say is the impact of leaving out this fix? Isn't that an accurate description? I don't know the current crash frequency so I didn't include that.

> So to clarify the impact from the Thunderbird perspective

I didn't ask for this on behalf of Thunderbird, but for Firefox. I actually don't have many clues how this impacts Thunderbird, other than what you're telling me here. My comments above is about what the impact on Firefox would be if we didn't merge the fix into those branches. I believe I'm following the correct procedure.
Comment on attachment 8906530 [details] [diff] [review]
v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch

This looks like a low volume crash on 56, but it might become worse in release. Fix verified on nightly in comment 62. Let's uplift this for beta 12.
Attachment #8906530 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment 68

8 months ago
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #67)
> This looks like a low volume crash on 56...

I haven't tried a 56 build, but in 55.0.3, the whole Firefox UI simply hangs instead of crashing the process, and the app must be killed.  This was first mentioned in comment 54; it might explain the drop in crash volume.

Comment 69

8 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/bd8b51b80738
status-firefox56: affected → fixed

Comment 70

8 months ago
badger, thank you for clarifying - I stand corrected. 

Unrelated - there can be other crash signatures. For example  EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER per bp-8559bf21-5ef4-4d54-ad50-a37d70170907 user comment "changing network" (same user sees  __pthread_kill | abort | free | _arc4_fork_child per bp-1ccd7603-5467-4e89-9985-9629f0170907 )
Comment on attachment 8906530 [details] [diff] [review]
v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch

this doesn't seem to be badly affecting esr52 so I'll skip this for esr52.4
Attachment #8906530 - Flags: approval-mozilla-esr52? → approval-mozilla-esr52-

Comment 72

8 months ago
(In reply to Julien Cristau [:jcristau] from comment #71)
> Comment on attachment 8906530 [details] [diff] [review]
> v2-0001-Bug-1359624-disable-nsAuthSambaNTLM-module-on-OSX.patch
> 
> this doesn't seem to be badly affecting esr52 so I'll skip this for esr52.4

I tested this bug on esr52 and it doesn't reproduce there using the same NTLM configuration where it reproduces on Firefox 55. So I think you're fine.
status-firefox-esr52: affected → wontfix
Uplifted for TB 52.4 ESR:
https://hg.mozilla.org/releases/mozilla-esr52/rev/2ddcbc54266b
status-thunderbird_esr52: affected → fixed
tracking-thunderbird_esr52: ? → 56+

Updated

8 months ago
Duplicate of this bug: 1353622

Comment 75

6 months ago
There is a very significant reduction in the vicinity of TB52.4.0 per graph for 
* libsystem_kernel.dylib@0x19dd6  - https://crash-stats.mozilla.com/signature/?product=Thunderbird&signature=libsystem_kernel.dylib%400x19dd6&date=%3E%3D2017-08-30T04%3A00%3A04.000Z&date=%3C2017-11-30T02%3A00%3A04.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#graphs
* __pthread_kill | abort | free | _arc4_fork_child - https://crash-stats.mozilla.com/signature/?signature=__pthread_kill%20%7C%20abort%20%7C%20free%20%7C%20_arc4_fork_child&date=%3E%3D2017-05-30T09%3A54%3A59.000Z&date=%3C2017-11-30T08%3A54%3A59.000Z#graphs
so the patch is fairly successful for Thunderbird

However, neither signature is totally gone.  __pthread_kill | abort | free | _arc4_fork_child for example ranks ~18 in Thunderbird Mac crashes, with an equal split between 10.12 and 10.13 users.   https://crash-stats.mozilla.com/signature/?product=Thunderbird&platform=Mac%20OS%20X&version=52.4.0&version=52.5.0&signature=__pthread_kill%20%7C%20abort%20%7C%20free%20%7C%20_arc4_fork_child&date=%3E%3D2017-08-30T05%3A59%3A01.000Z&date=%3C2017-11-30T03%3A59%3A01.000Z&_columns=date&_columns=product&_columns=version&_columns=reason&_columns=address&_columns=user_comments&_sort=-date&page=1  ...

for example bp-100cb3f2-e0b3-4ce2-9dae-296250171129

So might file some new bugs. I didn't check the Firefox side. Additional thoughts welcome
Status: RESOLVED → VERIFIED
(In reply to Wayne Mery (:wsmwk) from comment #75)
> However, neither signature is totally gone.  __pthread_kill | abort | free |
> _arc4_fork_child for example ranks ~18 in Thunderbird Mac crashes, with an
> equal split between 10.12 and 10.13 users.  
> https://crash-stats.mozilla.com/signature/
> ?product=Thunderbird&platform=Mac%20OS%20X&version=52.4.0&version=52.5.
> 0&signature=__pthread_kill%20%7C%20abort%20%7C%20free%20%7C%20_arc4_fork_chil
> d&date=%3E%3D2017-08-30T05%3A59%3A01.000Z&date=%3C2017-11-30T03%3A59%3A01.
> 000Z&_columns=date&_columns=product&_columns=version&_columns=reason&_columns
> =address&_columns=user_comments&_sort=-date&page=1  ...
> 
> for example bp-100cb3f2-e0b3-4ce2-9dae-296250171129
> 
> So might file some new bugs. I didn't check the Firefox side. Additional
> thoughts welcome
Flags: needinfo?(honzab.moz)

Comment 77

6 months ago
I'm quite certain that this bug is gone in Firefox 56 and higher, as I had an internal corporate website that I use daily and which I couldn't use with Firefox until the bug was fixed. Unfortunately I don't have a repro scenario for Thunderbird so can't say why the bug might be lingering there.
I looked at the crash search in comment 75 and there is nothing related to this particular ntlm bug.  If you feel a need to file new bugs, then feel free to do so, but I'm not even sure about a component to file them at.  All reports show just a weird stack:

https://crash-stats.mozilla.com/report/index/c6226076-003b-42fa-a79a-d94e90180109

Thread (0)

__pthread_kill
abort
free
_arc4_fork_child
_libc_fork_child
libSystem_atfork_child
fork
HIS_XPC_SetNetworkLocation
___ZL14SwitchLocationP13OpaqueMenuReft_block_invoke
_dispatch_call_block_and_release
_dispatch_client_callout
_dispatch_queue_override_invoke
_dispatch_root_queue_drain
_dispatch_worker_thread3
_pthread_wqthread
start_wqthread
dispatch_source_set_cancel_handler

all other threads are already gone.  according the uptime, this looks like a shutdown crash.
Flags: needinfo?(honzab.moz)
You need to log in before you can comment on or make changes to this bug.