Closed Bug 1722827 Opened 3 years ago Closed 2 years ago

Crash on macOS 12 Monterey Beta 4 With Unsupported Security Settings

Categories

(Core :: Widget: Cocoa, defect, P3)

Product:
Core
Component:
Widget: Cocoa
Version:
Firefox 90
Platform:
x86_64
macOS
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: leo.natan, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [mac:stability])

Crash Data

Attachments

(6 files)

firefox_2021-07-29-030347_Leo-Natans-MPB.crash
183.84 KB, text/plain
Details
Another crash dump
175.26 KB, text/plain
Details
Another crash dump, this time a different crash
149.14 KB, text/plain
Details
Nightly Crash Dump
82.12 KB, text/plain
Details
New profile crash
140.71 KB, text/plain
Details
Crash with new profile
140.71 KB, text/plain
Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62

Steps to reproduce:

There is a consistent and constant crash on macOS Monterey. It's always "Crashed Thread: 2 IPC I/O Parent"

Crashed Thread: 2 IPC I/O Parent

Exception Type: EXC_GUARD (SIGKILL)
Exception Codes: 0x0000000000020e03, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace <0x17>, Code 0x2000010000020e03

Thread 2 Crashed:: IPC I/O Parent
0 libsystem_kernel.dylib 0x00007ff80975ebba _kernelrpc_mach_port_deallocate_trap + 10
1 libsystem_kernel.dylib 0x00007ff80975fc82 mach_port_deallocate + 17
2 XUL 0x00000001291c7544 0x1288b9000 + 9495876

From another crash report:

Thread 6 Crashed:: IPC I/O Parent
0 libsystem_kernel.dylib 0x00007ff80975ebba _kernelrpc_mach_port_deallocate_trap + 10
1 libsystem_kernel.dylib 0x00007ff80975fc82 mach_port_deallocate + 17
2 XUL 0x0000000123111544 0x122803000 + 9495876

This is happening on Firefox and Firefox Beta channels.

Actual results:

Firefox crashes. See attached crash dump

Expected results:

Not crash

Attached file Another crash dump —

Hi Leo Natan,
Thanks for the report!
Can you attach your about:support information please?

Does this issue happen with a new profile? Here is a link on how to create a new profile: https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles

Are you using add-ons? If so could you please list them? (you can try the issue while in Safe Mode. You can find helpful info here : https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode .)

Does this issue occur in the latest nightly version of firefox? Here is a link from where you can download it: https://www.mozilla.org/en-US/firefox/channel/desktop/

I've assigned a component in order to get the dev team involved
Best regards, Clara.

Flags: needinfo?(leo.natan)
Component: Untriaged → Widget: Gtk
Product: Firefox → Core
Component: Widget: Gtk → Widget: Cocoa
OS: Unspecified → macOS
Hardware: Unspecified → x86_64

Leo, do you see any of these crashes listed in about:crashes? If so, could you post the link(s) here.

Have you only seen these with SIP disabled?

I don't have any insights into the crashes but here are some details picked out of the reports.

Hardware: MacBookPro15,1

Time Awake Since Boot: 1800 seconds
System Integrity Protection: disabled
Crashed Thread:        6  IPC I/O Parent (mach_port_deallocate on crashing thread stack)
Exception Type:        EXC_GUARD (SIGKILL)
Exception Codes:       0x000000000001f503, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY
Termination Reason:    Namespace <0x17>, Code 0x200001000001f503

Time Awake Since Boot: 68 seconds
System Integrity Protection: disabled
Crashed Thread:        6  IPC I/O Parent (mach_port_deallocate on crashing thread stack)
Exception Type:        EXC_GUARD (SIGKILL)
Exception Codes:       GUARD_TYPE_MACH_PORT
Exception Codes:       0x000000000001b803, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY
Termination Reason:    Namespace GUARD, Code 2305844108725434371 

Time Awake Since Boot: 3100 seconds
System Integrity Protection: disabled
Crashed Thread:        71  PortServerThread
Exception Type:        EXC_GUARD (SIGKILL)
Exception Codes:       GUARD_TYPE_MACH_PORT
Exception Codes:       0x000000000001be03, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY
Termination Reason:    Namespace GUARD, Code 2305844108725435907
Attached file Nightly Crash Dump —
Hi Clara, Nightly crashes consistently on launch for me. See attached crash dump. Completely unusable.

Hello Clara,

Nightly crashes consistently on launch for me. See attached crash dump. Completely unusable.

Oddly enough, after creating a new profile, I haven't been able to reproduce a crash. Will take my normal profile and pick it apart to try and narrow down what is causing the crash.

Haik, I always use my Mac with SIP and AMFI disabled (along with other hidden security gems). This is how I used it with Bug Sir, and how I continue to use it with Monterey. Not sure why that would cause crashes.

The crashes do not appear in about:crashes, only in ~/Library/Logs/DiagnosticReports/

Flags: needinfo?(leo.natan)

Clara, I spoke too soon. Right after posting the previous comment, I got the following crash with the new profile.

Attached file New profile crash —
Clara, I spoke too soon. Right after posting the previous comment, I got the following crash with the new profile.
Attached file Crash with new profile —

Willing to help debug this. If you want me to run a debug build, or whatever is needed, just let me know. Thanks

The crashes are all in mach_port_deallocate, in various parts of our SharedMemory infrastructure. For example, the crash from comment 6 (Nightly) is when deallocating the ports->mReceiver port on the PortServerThread: https://searchfox.org/mozilla-central/rev/525ff01397317b4b24b1a2cc51fe4511ec49ac8b/ipc/glue/SharedMemoryBasic_mach.mm#351

It looks like this macOS version changed something in the handling of mach_ports, either introducing a bug or revealing an existing bug.

Leo, could you report this bug to Apple via their Feedback app?

Sure thing, I opened FB9433749 for this issue.

I've reported this to our Apple contact, referencing FB9433749 and this bug.

It's concerning that we are not getting crash reports for this issue, but the problem might be occurring too early during startup or perhaps the ports issue affects crash reporting.

I have not been able to reproduce this on a MacBookPro15,1 with/without SIP.

See Also: → 1724215

I always use my Mac with SIP and AMFI disabled (along with other hidden security gems)

How do you disable AMFI? With the amfi_get_out_of_my_way=0x1 kernel boot arg? What about the other "hidden security gems"? What are they, and how do you disable them?

It's possible that disabling one or more of these triggers these crashes. At least we should try testing with them disabled. (I don't currently have a Monterey partition I can test with.)

Yes.

  • csrutil disable
  • sudo nvram boot-args="amfi_get_out_of_my_way=1"
  • sudo spctl --master-disable
  • sudo defaults write /Library/Preferences/com.apple.security.coderequirements Entitlements -string always
  • sudo defaults write /Library/Preferences/com.apple.security.coderequirements AllowUnsafeDynamicLinking -bool YES
  • sudo defaults write /Library/Preferences/com.apple.security.libraryvalidation.plist DisableLibraryValidation -bool YES

I don’t see an immediate or obvious suspect here.

Checked for any lingering kexts I might have. None, all Apple.

Leo, do you have any crashes at all in about:crashes? It's conceivable that Firefox's crash reporter has somehow gotten disabled on your system.

Doing kill -6 [firefox-pid] (sending a SIGABRT signal to the Firefox process) normally triggers the crash reporter. Does it do so on your system?

Bug 1724215 may explain why we don't see these crashes. i.e., we don't catch EXC_GUARD exceptions.

I have a crash report from February. Crash report is not disabled.

I have a crash report from February

But that must predate your Monterey upgrade :-)

Bug 1724215 may explain why we don't see these crashes. i.e., we don't catch EXC_GUARD exceptions.

I noticed. I just wanted to check other possibilities.

Why would the upgrade to 13.0 somehow disable crash reporting? Let’s please focus on the issue at hand. If there is a debug build with extended crash reporting, I’d be happy to test. Just please note that nightly currently cannot be used for long enough to get the crash information through the UI.

Leo, do please try kill -6 [firefox-pid]. But that's not actually a definitive test, because Breakpad (Firefox's crash handler) uses a signal handler to catch this kind of crash. To test the main functionality (which catches Mach exceptions), one needs to create a try build that will, say, do a NULL dereference on demand. I'll do one in the next few hours, and ask you to try that.

Yes, bug 1724215 may explain what's going on here. But it's a pretty far-out hypothesis. We should as least try to eliminate other possibilities.

Edit:

Just please note that nightly currently cannot be used for long enough to get the crash information through the UI.

I hadn't thought of that. Sigh.

I would like to concentrate debugging the crash itself, rather than the crash reporter. I understand the importance, just not what is my pressing issue. I am using MS Edge as least worst case in the mean time, I hope you understand my pain.

If you create a debug build, please start with the beta or production codebase, not nightly. Otherwise, I won’t be able to tell whether crash reporter worked. Nightly crashes on launch.

If you create a debug build, please start with the beta or production codebase, not nightly.

As best I can tell that's not possible. So I won't bother.

Are there no branches of the codebase? What if a hot fix was necessary?

@Leo, this does appear to be caused/triggered by your settings on comment 16. On my MacBookPro15,1 model, after enabling those settings and rebooting, I immediately encounter the same crash as you on Release and Nightly versions.

It would be great to narrow down which setting is causing the crash to see if we can understand what's causing this. But given that this is an unsupported configuration (for Apple), it may be tough to get support if this turns out to be an Apple issue.

Could you try and determine which setting triggers this? I could not reproduce this with just SIP disabled. With your settings from comment 16, it is not reproducible with a local build which differs in not being notarized or signed which suggests it is triggered by one of the com.apple.security settings.

--

Side note: for forcing crashes, we have about:crashparent and about:crashcontent pages for forcing crash reports in the parent and child process. Warning: they cause immediate crashes when loading the URL.

Yes, will do.

Seems related to amfi_get_out_of_my_way=1. Very bad news. I use that to disable code signing and entitlement validation.

Your comment gave me an idea. Removing the signature from Firefox.app using codesign --deep --remove-signature /Applications/Firefox.app fixes the crashes. So it must be some collision with some of the entitlements that Firefox uses.

Added the above information to the feedback report with Apple.

Adding any code signature, even without entitlements, using codesign -f --deep -s - /Applications/Firefox.app triggers the crashes. Apple has broken their OS completely, it seems.

(In reply to Leo Natan from comment #32)

Adding any code signature, even without entitlements, using codesign -f --deep -s - /Applications/Firefox.app triggers the crashes. Apple has broken their OS completely, it seems.

Note: using --deep is not a good way to sign Firefox.app because misses some files in the bundle. We have to walk to the list of files in the bundle and run codesign that way. I can provide some scripts if you're interested.

Do you see the same problem with any other apps?

It was a quick try. Since I have codesigning checking disabled (through amfi_get_out_of_my_way=1), the system doesn't actually verify signatures. At first, I was suspecting maybe one of the entitlements that come with Firefox, but noticed that even a simple signature reintroduces the crashing. Apple has broken something, or some issue has been exposed with Firefox.

No other app exhibits this issue. I assume most Apple software uses XPC for IPC, which uses mach ports internally, as does Safari (WebKit2): https://github.com/WebKit/WebKit/blob/main/Source/WebKit/Platform/cocoa/SharedMemoryCocoa.cpp, as well as Edge/Brave.

No such issues elsewhere.

Severity: -- → S4
Priority: -- → P3
Whiteboard: [mac:stability]
Status: UNCONFIRMED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Summary: Crash on macOS 12 Monterey Beta 4 → Crash on macOS 12 Monterey Beta 4 With Unsupported Security Settings

I'm leaving this bug open for now. Given that it only occurs with unsupported security settings, it is very low priority. But it would be beneficial to understand what is different about Firefox codesigning that triggers this problem in this configuration and possibly fix the problem.

Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: FIXED → ---
Crash Signature: [@_kernelrpc_mach_port_deallocate_trap]

Product Firefox
Release Channel nightly
Version 95.0a1
Build ID 20211028094021 (2021-10-28) Buildhub data
OS macOS 12
OS Version 12.0.1 21A559

bp-3838b3cc-7b52-43d3-8a14-26f020211028 = Uptime 45 seconds
bp-0227ede7-5051-46c9-ab6b-efe7c0211028 = Uptime 242 seconds (4 minutes and 2 seconds)

Signature: _kernelrpc_mach_port_deallocate_trap

Crash Reason EXC_GUARD / GUARD_TYPE_MACH_PORT / GUARD_EXC_INVALID_RIGHT

This is for Sumo's Related Bugs
1722827 REOPENED --- Crash on macOS 12 Monterey Beta 4 With Unsupported Security Settings

Facing the same issue on macOS Monterey 12.0.1 (stable - Apple M1) with latest Firefox 93.0.
Refresh and reinstallation didn't solve. The crash either occurs at launch or randomly.

This is likely due to some changes on macOS SIP, namely after running the following commands:

csrutil enable --without kext --without fs --without nvram
nvram boot-args="amfi_get_out_of_my_way=1"

(In reply to Leo Natan from comment #30)

Your comment gave me an idea. Removing the signature from Firefox.app using codesign --deep --remove-signature /Applications/Firefox.app fixes the crashes. So it must be some collision with some of the entitlements that Firefox uses.

I confirm this does seem to fix the crashes

Interesting, this issue does not reproduce on Apple M1 Max CPU, only on Intel.

Closing because no crashes reported for 12 weeks.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 2 years ago
Resolution: --- → WORKSFORME
Duplicate of this bug: 1837274
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: