1578497 - PKG for macOS needs to be notarized

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Signing, defect)

Description

[Mike Kaply [:mkaply]]

Come to find out, even though the app in the PKG is notarized, the PKG needs to be notarized as well.

It's not installing on Catalina.

Comment 1

[Aki Sasaki (not active)]

11:05 <aki> mkaply: did someone verify that it is installing on mojave?
11:05 <aki> ruling out a broken pkg file
11:06 <mkaply> aki: Yep, works there
11:06 <aki> ok
11:07 <aki> i wonder if we can create the installer pre-notarization, so we only have to notarize once

Comment 2

[Mike Kaply [:mkaply]]

11:07 <aki> i wonder if we can create the installer pre-notarization, so we only have to notarize once

Docs seem to indicate yet:

"You can notarize an existing disk image, installer package, or ZIP archive containing your app."

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Mike, what's the timeline for this ? How does that relate to Catalina shipping ? We'll have to figure out how to slot this into existing work, and I don't know how many enterprise folks are already using these pkgs need Catalina support.

Some notes from talking to Aki:

• we prefer not to sequentially notarize the app, create the package, then notarize the package, because it'll be a time hit and reduce throughput
• we should try notarizing a pkg built from an unnotarized app, and see if it launches (comment #2). And the same for a notarized app. I'll create some experimental pkg's for testing
• we might add a new behaviour to iscript which handles pkg creation and notarization separately from dmg. We might use that to split out pkg generation in the taskgraph, which could get dmgs/updates to QE earlier, but overall we'd still depend on pkgs for the checksums files

Comment 4

[Mike Kaply [:mkaply]]

Catalina is probably going to be released this Monday (there's an Apple event).

macOS users tend to upgrade faster than other OSes, but I do suspect that Catalina will take longer.

So the answer is as soon as we can. I've already gotten feedback from admins that they won't use anything that's not notarized at this point.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

OK. History says macOS releases are later in September but I understand the feedback about wanting notarization. I'll see what I can do about slotting this in as soon as I can.

Some preliminary investigations - if I sign a Firefox app, build a signed pkg, notarize only the pkg, and install the pkg on Catalina beta 7, then there are no errors install or first run. This is despite the app inside the pkg not being notarized (ie no Contents/CodeResources is present), which might have something to do with not quarantine attribute not being set on the app. This may allow us to notarize the pkg relatively efficiently in the current automation.

One downside is that a pkg install will fail the first partial update, because it will want to patch Contents/CodeResources. I think we can work around this building the partial to overwrite instead (ADD vs PATCH instructions in the manifest). We could also resolve bug 1473751 if we use the same approach on Contents/_CodeSignature/CodeResources.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Apple has announced Catalina will be released at some point in October.

I've made some progress on notarizing the pkg with

• https://github.com/nthomas-mozilla/scriptworker-scripts/compare/master...notarize-pkg
• https://hg.mozilla.org/try/rev/818f0830b891e891e5a3d4afbfdfbfd20ada12f7
• example log for 4 locales

That sends the signed app and pkgs to Apple in the same zip file, and both are notarized separately (like comment #5). The example log took just under 20 minutes for the 4 locales, which should be compared to 13-27 min for 70.0b5 tasks handling 5 locales. I need to run more tests on timing but at first glance it seems this approach could work.

Comment 7

[Aki Sasaki (not active)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #6)

That sends the signed app and pkgs to Apple in the same zip file, and both are notarized separately (like comment #5). The example log took just under 20 minutes for the 4 locales, which should be compared to 13-27 min for 70.0b5 tasks handling 5 locales. I need to run more tests on timing but at first glance it seems this approach could work.

\o/

async def create_all_notarization_zipfiles(all_paths, path_attr=["app_path"]):

Lists and dicts as kwarg values is a python footgun.

>>> def foo(num, a=[1]):
...     a.append(num)
...     print(a)
...
>>> foo(2)
[1, 2]
>>> foo(3)
[1, 2, 3]
>>> foo(4)
[1, 2, 3, 4]

We should probably make this a tuple, or force callers to specify the full path_attrs without a default, or only specify the default if path_attr is None (and make it None by default).

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

I've integrated that fix, and have been doing some extra testing. Getting an update to apply to staging releases when elevating privileges is a bit tricky.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: GitHub Pull Request - scriptworker-scripts

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1578497 - Overwrite notarization ticket on each partial update instead of patching, so pkgs can be updated, r=bhearsum

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Testing details - some 71.0bN releases on try, last of which is here

• packages install on Catalina now, not just Mojave. No prompts seen on first run
• signing works OK
  • vanilla and partner, en-US and multi-builds
  • can't detect any change in elapsed job time (within the natural variation of external services)
  • logs are more readable with the fix to skip signing inner apps, and no more verbose stapling
• update verify works, but doesn't test the partial case. I did that separately with a custom verify config and a docker image. Also tested new pkg + new complete update == new dmg

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ed2d14a314f5
Overwrite notarization ticket on each partial update instead of patching, so pkgs can be updated, r=bhearsum

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

I've deployed the github changes to the prod signing hardware, so we'll get notarized packages from the next nightly, beta, and release.

The gecko change will merge to m-c at some point in the next day, and will need uplifting to beta/release/esr68. I'll verify updates for pkgs using the second nightly built with it.

Comment 14

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/ed2d14a314f5

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

I've successfully installed the Nightly en-US pkg on Catalina with buildID 20191001041624, and updated to 20191001213452 via partial update. The partial generation log has
2019-10-01 23:08:08,930 - WARNING - target.partial-1.mar: add "Contents/CodeResources" (forced)

Uplifted the gecko change to the release branches:
https://hg.mozilla.org/releases/mozilla-beta/rev/b07a932f6af3d95c5429c3051d625ce9b9ff1556
https://hg.mozilla.org/releases/mozilla-release/rev/e40b66e55c283267a74ea411b9dfeeb7ddbb879c
https://hg.mozilla.org/releases/mozilla-esr68/rev/4cfa0d6f216a44535f0abd372b39a6b234372c6c