Closed Bug 1470607 Opened Last year Closed 3 months ago

[macOS 10.14] Use the Apple Notary Service for signed builds

Categories

(Firefox Build System :: General, defect)

Unspecified
macOS
defect
Not set

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1471004

People

(Reporter: haik, Unassigned)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

Attachments

(4 files, 3 obsolete files)

Apple recently announced its Notary Service, a new signing service for apps delivered outside of the App Store. The service offers "App Notarization" where submitted applications are checked for malware by Apple using automated security checks. The first-run user experience is slightly better for an app that has been notarized with the service. From the information available[1], the difference is that the first-run dialog includes the application dialog and not the cautious dialog displayed now. See attached screenshot.

This requires the app opting in to the new 10.14 enhanced runtime[2].

1. https://developer.apple.com/videos/play/wwdc2018/702/
2. Bug 1470597 - Investigate enabling Enhanced Runtime for the parent process
Depends on: 1470597
See Also: → mojave
Blocks: mojave
See Also: mojave
See Also: → 1471004
Depends on: mojave-sdk
I was going to move this bug to Release Engineering, but I see that kmoir has filed bug 1471004. I'm not sure what build system work this would entail--AFAIK all the signing bits are done external to the build system these days.
Apple has now officially stated[1] that signing with the Notary Service will be required in a future release. "In an upcoming release of macOS, Gatekeeper will require Developer ID–signed software to be notarized by Apple." This could mean that an app that isn't notarized will require a right-click open like a downloaded unsigned app now.

1. https://developer.apple.com/news/?id=10192018a&1539965082
Depends on: 1471004
No longer depends on: mojave-sdk
Removed the dependency on bug 1475652 (build with the 10.14 SDK). Using the Notary Service does require opting into Enhanced Runtime, but Apple confirmed enabling Enhanced Runtime doesn't require building with the 10.14 SDK. It does require running the codesign command on 10.13.6 or newer.
Ignoring the crashing problem described on bug 1470597 comment 12, I'm able to successfully notarize a release build (thanks to a tip from :spohl) by copying it from /Applications/Firefox.app and running codesign commands. There seems to be an ordering dependency when signing binaries within the .app where some binaries must be signed before some others. As a quick hack, the commands below run codesign on some binaries and then use the find command sign all the binaries.

  $ cp -r /Applications/Firefox.app .

  $ codesign --force -o runtime --verbose --sign "D2C9222U5Z" \
  ./Firefox.app/Contents/MacOS/XUL \
  ./Firefox.app/Contents/MacOS/pingsender \
  ./Firefox.app/Contents/MacOS/*.dylib \
  ./Firefox.app/Contents/MacOS/crashreporter.app/Contents/MacOS/minidump-analyzer \
  ./Firefox.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter \
  ./Firefox.app/Contents/MacOS/firefox-bin \
  ./Firefox.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container \
  ./Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater \
  ./Firefox.app/Contents/MacOS/firefox

  $ find ./Firefox.app -type f -exec codesign --force -o runtime --verbose --sign "D2C9222U5Z" {} \;

  $ codesign --force -o runtime --verbose --sign "D2C9222U5Z" Firefox.app

  $ zip -r Firefox.app.zip Firefox.app

  $ xcrun altool --notarize-app --primary-bundle-id "org.haikaftandilian.firefox" \
     --username <apple-developer-ID> --password "@keychain:AC_PASSWORD" --file Firefox.app.zip

OOC, how long does that final 'xcrun' command take? Does it modify Firefox.app.zip directly?

(In reply to Chris AtLee [:catlee] from comment #6)

OOC, how long does that final 'xcrun' command take? Does it modify Firefox.app.zip directly?

It takes 2-3 minutes for me with a zip file that is 76MB. It doesn't modify the zip file.

That is not the time it takes for the approval to occur, just to get the request submitted.

I just did a test and the zip file was listed as accepted in less than two minutes after the --notarize-app command completed.

There's one more command that needs to be run after the acceptance which completes within a few seconds:

$ xcrun stapler staple -v Nightly.app

I'm not hitting the crash I mentioned in Comment 5. It was just a matter of setting the appropriate entitlements. I'm working on determining what entitlements we would want to use for Firefox builds and plan to land an entitlement file that releng would use when running the codesign and xcrun commands.

Nick and I are looking at automating this; first we'll need to successfully notarize manually.

Did you set up a developer account? We may do so for testing, to avoid using the production account.
Did you recompile, or just specify a different --primary-bundle-id to test?
Do we need new builds every time we re-submit for testing? (We're guessing re-signing may suffice, since that will change the contents, but we're not sure.)

(Just asking to save time if you already have the answers here.)

A hacky bash script for testing Notarization:

$ ./codesign-bundle-hardened.bash
Usage: ./codesign-bundle-hardened.bash
-a <PATH-TO-BROWSER.app>
-i <IDENTITY>
-e <ENTITLEMENTS-FILE>
[-o <OUTPUT-ZIP-FILE>]

$ ./codesign-bundle-hardened.bash -a ./Nightly.app
-i <YOUR_SIGNING_IDENTITY>
-e ./browser.entitlements
-o Nightly.app.zip

$ xcrun altool --notarize-app
-f Nightly.app.zip
--primary-bundle-id "org.haikaftandilian.nightly"
-u <APPLE_DEVELOPER_ID> # your apple developer ID email address
-p "@keychain:AC_PASSWORD"

Wait for email regarding notarization acceptance

$ xcrun stapler staple -v Nightly.app

Attachment #9035236 - Attachment description: Hardened Runtime entitlements for signing app for Notarization → browser.entitlements.txt Hardened Runtime entitlements file for signing app for Notarization

(In reply to Aki Sasaki [:aki] from comment #9)

Nick and I are looking at automating this; first we'll need to successfully notarize manually.

Did you set up a developer account? We may do so for testing, to avoid using the production account.

I already had a developer account and followed some steps online to setup the cert. I know it's possible to create alternate (e.g., developer or testing) certificates for the same account so you could use a new testing cert on the production account, but I think you're saying you'd rather avoid using the production account entirely.

Did you recompile, or just specify a different --primary-bundle-id to test?

I've tested with my own build from central and also with re-signing an official release build.

Do we need new builds every time we re-submit for testing? (We're guessing re-signing may suffice, since that will change the contents, but we're not sure.)

I've been resubmitting the same build after re-running the codesign commands. My script uses the codesign --force option to overwrite existing signatures. I've found that submitting a duplicate zip file is detected and blocked, but after re-running the codesign commands the zip file is not identical.

I tried notarizing an existing signed Firefox Nightly dmg. We have a bunch of errors, but they fall into 3 buckets:

  • The executable does not have the hardened runtime enabled. These pages describe how to fix this. It looks like bug 1470597 is hard-blocking the notarization work?
    • (I did download this binary from Dec 31; if we've landed the hardened runtime changes since then, I can retry with a newer binary.)
  • The signature algorithm used is too weak. I'm not sure about this -- we're using current Developer IDs. It's possible our mac signing servers are using an older XCode or are configured to use an older algorithm; we'll have to dig.
  • The binary is not signed. We have rules on the signing server about which binaries to skip; we can probably sign this binary as well.

(I know there's instructions above on how to sign the app before notarizing, but ideally we'd be able to keep our existing signing automation and just add the notarization piece. It looks like we have to at least adjust our signing for this to work.)

(In reply to Haik Aftandilian [:haik] from comment #4)

Removed the dependency on bug 1475652 (build with the 10.14 SDK). Using the
Notary Service does require opting into Enhanced Runtime, but Apple
confirmed enabling Enhanced Runtime doesn't require building with the 10.14
SDK. It does require running the codesign command on 10.13.6 or newer.

Doh. Our signing servers are 10.10.5.
We can either upgrade the mac signing server pool (or spin up a 2nd mac signing server pool and deprecate the original), or sign on the same macs that run the notarization steps.

I seem to recall hitting problems the last time we wanted to update which version of macOS the signing servers were running on. Something about compatibility for the signed binaries on older versions of macOS when they've been signed on newer versions of macOS.

According to https://www.mozilla.org/en-US/firefox/64.0.2/system-requirements/, we still support 10.9, so we should make sure that the new signed/notarized binaries still work properly on 10.9 systems.

From some brief research it sounds like if we sign on a newer version of macOS but continue to build with -mmacosx-version-min=10.9 we should be OK:

http://www.discretecosine.com/index-html-2/
https://stackoverflow.com/questions/41865537/how-does-apples-codesign-utility-decide-which-sha-algorithms-to-sign-a-shared

codesign will include both version 1 and 2 signatures if targeting macOS earlier than 10.10, and when targeting versions older than 10.12 codesign will include both sha-1 and sha-256 hashes in the code signature.

Using the Notary Service requires that the app not be codesigned with the com.apple.security.get-task-allow entitlement and this in turn prevents users from attaching a debugging to any of the browser processes. The workaround is to disable system integrity protection (SIP). I'm going to send a mail to dev-platform to get some feedback on this. If our developers want to continue to be able to debug builds on the Nightly channel (without the workaround) we will have to skip notarization on that channel.

Aki or Ted, if you're familiar with this, could you confirm when we run codesign today? For example, is it just on official channel builds and not for any local or try builds? Once we start using the notary service, the entitlements we use to codesign the binaries will affect execution in some ways. Initially, the only significant effect will be the ability to attach a debugger, but in the future we may want local/try builds to be codesigned with certain entitlements so those builds match what we ship to users (apart from allowing the debugger.)

Flags: needinfo?(ted)
Flags: needinfo?(aki)

We codesign in multiple places/branches, mainly for l10n-on-push iirc (l10n repacks repack signed binaries).

However, we have 3 levels of signing: release signing, nightly signing, and depend signing. (depend signing doesn't ship anywhere, and is only there for test/development purposes.) I believe the only difference today is which key they sign with, but I think we could add configuration to only notarize on release, or on release and nightly, as needed.

Flags: needinfo?(aki)

Attaching new screenshot comparing notarized first run dialog to not-notarized. When Apple announced this at WWDC the notarized first run dialog included the App icon, as shown in the earlier attached screenshot[1]. I'm not seeing that in my tests and will try to resolve this. If this has changed, then the earlier screenshot is now obsolete.

  1. https://bugzilla.mozilla.org/attachment.cgi?id=8987232&action=edit
Attachment #8987232 - Attachment is obsolete: true

Image was edited to replace an internal IP address with "www.mozilla.org" on the notarized dialog.

Attachment #9036014 - Attachment is obsolete: true
Depends on: 1522409

Sounds like Aki answered your question. (FWIW I don't really know anything about our code signing process.)

Flags: needinfo?(ted)
Depends on: 1528087

Rob, I'll add you to this bug as Thunderbird will need this too.

Depends on: 1543830
See Also: → 1556733
Blocks: catalina
Depends on: 1559274

Should we resolve this bug too?

Yes. Closing as a dupe of 1471004.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1471004
No longer depends on: 1528087
You need to log in before you can comment on or make changes to this bug.