Closed Bug 1689807 Opened 26 days ago Closed 25 days ago

macOS startup crash "Code Signature Invalid"

Categories

(Firefox Build System :: General, defect)

Unspecified
macOS
defect

Tracking

(firefox-esr78 unaffected, firefox84 unaffected, firefox85 unaffected, firefox86 unaffected, firefox87blocking fixed)

RESOLVED FIXED
87 Branch
Tracking Status
firefox-esr78 --- unaffected
firefox84 --- unaffected
firefox85 --- unaffected
firefox86 --- unaffected
firefox87 blocking fixed

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: crash)

Attachments

(1 file)

We are are seeing crash reports on macOS, starting with Nightly 87.0a1 20210130054348. Nightly updates have been halted.

Sören provided a screenshot which mentions "Code Signature Invalid". Both Sören and jandem mention it getting reported as JS engine crash.

It already crashes when Jan tries to launch the profiler picker.

Mike, can you check if there are issues with the builds (e.g. with the signature), please?

Last good to first bad nightly autoland build based on the affected Nightly builds would be: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=repack&tochange=059539aeb4852ef5b7f37f578ac510dffa1056fe&fromchange=8caecd73211a92ce369523ab42b637b851b41742

Flags: needinfo?(mh+mozilla)
Crash Signature: [@ EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER] [@ EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #0)

It already crashes when Jan tries to launch the profiler picker but the crash reports 'Crash annotations' tab always mention profile data like Add-ons installed (and always some add-ons installed by the user).

The crashing process gets reported as rdd.

I actually don't know if these crash reports are related to this. The crash I'm seeing occurs very early in startup and I don't see any crash reports being sent to Mozilla, just Apple's crash report thing.

When I run Firefox from the command line, I get this:

dyld: Library not loaded: @executable_path/libnss3.dylib
  Referenced from: /Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs
dyld: Library not loaded: @executable_path/libnss3.dylib
  Referenced from: /Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs

When I tried to turn off System Integrity Protection to try to debug more, it didn't crash anymore.

Crash Signature: [@ EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER] [@ EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST]
Summary: macOS startup crash in [@ EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER] and [@ EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST] → macOS startup crash "Code Signature Invalid"

Information in comment 0 have been updated.

Bug 1689232 was part of the last merge and talks about mac signing changes. Could it be the cause?

(In reply to Ted Campbell [:tcampbell] from comment #4)

Bug 1689232 was part of the last merge and talks about mac signing changes. Could it be the cause?

I agree that one seems most likely. Maybe we can try backing out from central?

It looks like the broken build has no entitlements:

$ codesign -d --entitlements - /Volumes/Firefox\ Nightly\ 1/Firefox\ Nightly.app
Executable=/Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/firefox

The crash under EnterJit from Apple's crash report then makes sense, we're not allowed to execute (unsigned) JIT code.

Printing entitlements for a 'good' build on the other hand has this:

$ codesign -d --entitlements - /Volumes/Firefox\ Nightly\ 2/Firefox\ Nightly.app
Executable=/Volumes/Firefox Nightly 2/Firefox Nightly.app/Contents/MacOS/firefox
??qH<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<!--
     Entitlements to apply to the .app bundle and all executable files
     contained within it during codesigning of production channel builds that
     will be notarized. These entitlements enable hardened runtime protections
     to the extent possible for Firefox.
-->
<plist version="1.0">
  <dict>
    <!-- Firefox does not use MAP_JIT for executable mappings -->
    <key>com.apple.security.cs.allow-jit</key><false/>

    <!-- Firefox needs to create executable pages (without MAP_JIT) -->
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/>
... more of the same ...
Duplicate of this bug: 1689811

Bug 1689232 has been backed out. This bug will get updated once the builds are complete and updates have been reenabled.

Nightly updates have been turned on.

Flags: needinfo?(mh+mozilla)
Status: NEW → RESOLVED
Closed: 25 days ago
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch

Exactly the same startup crash happens for Thunderbird Daily as well (still happens in build from 2021-02-01). Is there something the Thunderbird people have to know? If so could you contact the appropriate people?

Flags: needinfo?(rob)

I'm tracking this in bug 1690118 for Thunderbird. Though I suspect that we have other issues as well.

Flags: needinfo?(rob)

Rob, this issue is not about Apple Silicon, it affects Intel Macs as well, and in bug 1690118 you said it "only cause issues when running on aarch64 hardware". So I wonder if bug 1690118 is really about the same issue.

Flags: needinfo?(rob)

It's the same issue on both, and is unrelated to this bug, but Thunderbird probably did hit this one as well. I just found it on Silicon first.

Flags: needinfo?(rob)

For posterity, I dug into this more, and I'm pretty certain this was caused by not signing with entitlements for the overall app bundle (the inner files still had entitlements applied to them).

You need to log in before you can comment on or make changes to this bug.