Closed Bug 1578497 Opened 7 months ago Closed 6 months ago

PKG for macOS needs to be notarized

Categories

(Release Engineering :: Release Automation: Signing, defect)

defect
Not set

Tracking

(firefox-esr60 unaffected, firefox-esr68 fixed, firefox69 fixed, firefox70 fixed, firefox71 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- fixed
firefox69 --- fixed
firefox70 --- fixed
firefox71 --- fixed

People

(Reporter: mkaply, Assigned: nthomas)

References

Details

Attachments

(2 files)

Come to find out, even though the app in the PKG is notarized, the PKG needs to be notarized as well.

It's not installing on Catalina.

11:05 <aki> mkaply: did someone verify that it is installing on mojave?
11:05 <aki> ruling out a broken pkg file
11:06 <mkaply> aki: Yep, works there
11:06 <aki> ok
11:07 <aki> i wonder if we can create the installer pre-notarization, so we only have to notarize once

11:07 <aki> i wonder if we can create the installer pre-notarization, so we only have to notarize once

Docs seem to indicate yet:

"You can notarize an existing disk image, installer package, or ZIP archive containing your app."

Assignee: nobody → aki
Component: General → Release Automation: Signing
Product: Firefox Build System → Release Engineering
QA Contact: aki

Mike, what's the timeline for this ? How does that relate to Catalina shipping ? We'll have to figure out how to slot this into existing work, and I don't know how many enterprise folks are already using these pkgs need Catalina support.

Some notes from talking to Aki:

  • we prefer not to sequentially notarize the app, create the package, then notarize the package, because it'll be a time hit and reduce throughput
  • we should try notarizing a pkg built from an unnotarized app, and see if it launches (comment #2). And the same for a notarized app. I'll create some experimental pkg's for testing
  • we might add a new behaviour to iscript which handles pkg creation and notarization separately from dmg. We might use that to split out pkg generation in the taskgraph, which could get dmgs/updates to QE earlier, but overall we'd still depend on pkgs for the checksums files
Flags: needinfo?(mozilla)
Assignee: aki → nthomas

Catalina is probably going to be released this Monday (there's an Apple event).

macOS users tend to upgrade faster than other OSes, but I do suspect that Catalina will take longer.

So the answer is as soon as we can. I've already gotten feedback from admins that they won't use anything that's not notarized at this point.

Flags: needinfo?(mozilla)

OK. History says macOS releases are later in September but I understand the feedback about wanting notarization. I'll see what I can do about slotting this in as soon as I can.

Some preliminary investigations - if I sign a Firefox app, build a signed pkg, notarize only the pkg, and install the pkg on Catalina beta 7, then there are no errors install or first run. This is despite the app inside the pkg not being notarized (ie no Contents/CodeResources is present), which might have something to do with not quarantine attribute not being set on the app. This may allow us to notarize the pkg relatively efficiently in the current automation.

One downside is that a pkg install will fail the first partial update, because it will want to patch Contents/CodeResources. I think we can work around this building the partial to overwrite instead (ADD vs PATCH instructions in the manifest). We could also resolve bug 1473751 if we use the same approach on Contents/_CodeSignature/CodeResources.

Apple has announced Catalina will be released at some point in October.

I've made some progress on notarizing the pkg with

That sends the signed app and pkgs to Apple in the same zip file, and both are notarized separately (like comment #5). The example log took just under 20 minutes for the 4 locales, which should be compared to 13-27 min for 70.0b5 tasks handling 5 locales. I need to run more tests on timing but at first glance it seems this approach could work.

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #6)

That sends the signed app and pkgs to Apple in the same zip file, and both are notarized separately (like comment #5). The example log took just under 20 minutes for the 4 locales, which should be compared to 13-27 min for 70.0b5 tasks handling 5 locales. I need to run more tests on timing but at first glance it seems this approach could work.

\o/

async def create_all_notarization_zipfiles(all_paths, path_attr=["app_path"]):

Lists and dicts as kwarg values is a python footgun.

>>> def foo(num, a=[1]):
...     a.append(num)
...     print(a)
...
>>> foo(2)
[1, 2]
>>> foo(3)
[1, 2, 3]
>>> foo(4)
[1, 2, 3, 4]

We should probably make this a tuple, or force callers to specify the full path_attrs without a default, or only specify the default if path_attr is None (and make it None by default).

I've integrated that fix, and have been doing some extra testing. Getting an update to apply to staging releases when elevating privileges is a bit tricky.

Testing details - some 71.0bN releases on try, last of which is here

  • packages install on Catalina now, not just Mojave. No prompts seen on first run
  • signing works OK
    • vanilla and partner, en-US and multi-builds
    • can't detect any change in elapsed job time (within the natural variation of external services)
    • logs are more readable with the fix to skip signing inner apps, and no more verbose stapling
  • update verify works, but doesn't test the partial case. I did that separately with a custom verify config and a docker image. Also tested new pkg + new complete update == new dmg
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ed2d14a314f5
Overwrite notarization ticket on each partial update instead of patching, so pkgs can be updated, r=bhearsum

I've deployed the github changes to the prod signing hardware, so we'll get notarized packages from the next nightly, beta, and release.

The gecko change will merge to m-c at some point in the next day, and will need uplifting to beta/release/esr68. I'll verify updates for pkgs using the second nightly built with it.

Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED

I've successfully installed the Nightly en-US pkg on Catalina with buildID 20191001041624, and updated to 20191001213452 via partial update. The partial generation log has
2019-10-01 23:08:08,930 - WARNING - target.partial-1.mar: add "Contents/CodeResources" (forced)

Uplifted the gecko change to the release branches:
https://hg.mozilla.org/releases/mozilla-beta/rev/b07a932f6af3d95c5429c3051d625ce9b9ff1556
https://hg.mozilla.org/releases/mozilla-release/rev/e40b66e55c283267a74ea411b9dfeeb7ddbb879c
https://hg.mozilla.org/releases/mozilla-esr68/rev/4cfa0d6f216a44535f0abd372b39a6b234372c6c

You need to log in before you can comment on or make changes to this bug.