1689807 - macOS startup crash "Code Signature Invalid"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

We are are seeing crash reports on macOS, starting with Nightly 87.0a1 20210130054348. Nightly updates have been halted.

Sören provided a screenshot which mentions "Code Signature Invalid". Both Sören and jandem mention it getting reported as JS engine crash.

It already crashes when Jan tries to launch the profiler picker.

Mike, can you check if there are issues with the builds (e.g. with the signature), please?

Last good to first bad nightly autoland build based on the affected Nightly builds would be: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=repack&tochange=059539aeb4852ef5b7f37f578ac510dffa1056fe&fromchange=8caecd73211a92ce369523ab42b637b851b41742

Comment 1

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

Attachment: screenshot crash

Comment 2

[Jan de Mooij [:jandem]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #0)

It already crashes when Jan tries to launch the profiler picker but the crash reports 'Crash annotations' tab always mention profile data like Add-ons installed (and always some add-ons installed by the user).

The crashing process gets reported as rdd.

I actually don't know if these crash reports are related to this. The crash I'm seeing occurs very early in startup and I don't see any crash reports being sent to Mozilla, just Apple's crash report thing.

When I run Firefox from the command line, I get this:

dyld: Library not loaded: @executable_path/libnss3.dylib
  Referenced from: /Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs
dyld: Library not loaded: @executable_path/libnss3.dylib
  Referenced from: /Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs

When I tried to turn off System Integrity Protection to try to debug more, it didn't crash anymore.

Comment 3

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

Information in comment 0 have been updated.

Comment 4

[Ted Campbell [:tcampbell]]

Bug 1689232 was part of the last merge and talks about mac signing changes. Could it be the cause?

Comment 5

[Jan de Mooij [:jandem]]

(In reply to Ted Campbell [:tcampbell] from comment #4)

Bug 1689232 was part of the last merge and talks about mac signing changes. Could it be the cause?

I agree that one seems most likely. Maybe we can try backing out from central?

Comment 6

[Jan de Mooij [:jandem]]

It looks like the broken build has no entitlements:

$ codesign -d --entitlements - /Volumes/Firefox\ Nightly\ 1/Firefox\ Nightly.app
Executable=/Volumes/Firefox Nightly 1/Firefox Nightly.app/Contents/MacOS/firefox

The crash under EnterJit from Apple's crash report then makes sense, we're not allowed to execute (unsigned) JIT code.

Printing entitlements for a 'good' build on the other hand has this:

$ codesign -d --entitlements - /Volumes/Firefox\ Nightly\ 2/Firefox\ Nightly.app
Executable=/Volumes/Firefox Nightly 2/Firefox Nightly.app/Contents/MacOS/firefox
??qH<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<!--
     Entitlements to apply to the .app bundle and all executable files
     contained within it during codesigning of production channel builds that
     will be notarized. These entitlements enable hardened runtime protections
     to the extent possible for Firefox.
-->
<plist version="1.0">
  <dict>
    <!-- Firefox does not use MAP_JIT for executable mappings -->
    <key>com.apple.security.cs.allow-jit</key><false/>

    <!-- Firefox needs to create executable pages (without MAP_JIT) -->
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/>
... more of the same ...

Comment 8

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

Bug 1689232 has been backed out. This bug will get updated once the builds are complete and updates have been reenabled.

Comment 9

[Sebastian Hengst [:aryx] (needinfo on intermittent or backout)]

Nightly updates have been turned on.

Comment 10

[Sören Hentzschel]

Exactly the same startup crash happens for Thunderbird Daily as well (still happens in build from 2021-02-01). Is there something the Thunderbird people have to know? If so could you contact the appropriate people?

Comment 11

[Rob Lemley [:rjl]]

I'm tracking this in bug 1690118 for Thunderbird. Though I suspect that we have other issues as well.

Comment 12

[Sören Hentzschel]

Rob, this issue is not about Apple Silicon, it affects Intel Macs as well, and in bug 1690118 you said it "only cause issues when running on aarch64 hardware". So I wonder if bug 1690118 is really about the same issue.

Comment 13

[Rob Lemley [:rjl]]

It's the same issue on both, and is unrelated to this bug, but Thunderbird probably did hit this one as well. I just found it on Silicon first.

Comment 14

[[e-mail] (:bhearsum)]

For posterity, I dug into this more, and I'm pretty certain this was caused by not signing with entitlements for the overall app bundle (the inner files still had entitlements applied to them).