1774273 - Transition macOS Notarization from altool to rcodesign

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Signing, enhancement)

Description

[Haik Aftandilian [:haik]]

At WWDC22, in What’s new in notarization for Mac apps[1], Apple announced that notarization requests must transition from using the CLI tool altool to its replacement notarytool. And they have announced "notarization with all forms of altool will stop working in fall 2023".

In the same presentation, Apple has announced a new REST API for Notarization with webhook support for getting status.

The new tool notarytool was introduced at WWDC21 in Faster and simpler notarization for Mac apps[2].

1. https://developer.apple.com/videos/play/wwdc2022/10109/
2. https://developer.apple.com/videos/play/wwdc2021/10261/

Comment 1

[Aki Sasaki (not active)]

From the video: notarytool submit path/to/zip --wait ; 98% within 15 min, most within 5. Or instead of --wait, --webhook URL.
notarytool log $SUBMISSION_ID notary-log.json which we could upload as an artifact.

There's also https://mozilla-hub.atlassian.net/browse/RELENG-867 ; the move to REST APIs may allow us to directly call the API via aiohttp and not need to be on a mac or iTMSTransporter. We'll need to investigate.

Comment 2

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Add mac notarization on signingscript r?bhearsum

Created a separate kind for only signing with iscript, and another for notarization.
Once we validate this is good, we need to add it to l10n/emfree/etc and point repackage kind at it.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/84a4ee59afa0
Add mac notarization on signingscript r=bhearsum

Comment 4

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/84a4ee59afa0

Comment 5

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Fix new mac notarization tasks r?bhearsum

Missed a scenario for the signing task behavior. Unfortunately it's only really testable in production.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d5260d63a651
Fix new mac notarization tasks r=bhearsum

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/d5260d63a651

Comment 8

[Heitor Neiva [:hneiva]]

The last change broke the signing task. Fix coming up shortly.

Comment 10

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Fix mac signing transform overrides r?#releng

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/54a722e614e6
Fix mac signing transform overrides r=releng-reviewers,gbrown

Comment 12

[Intermittent Failures Robot]

14 failures in 3863 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12
• try: 2

Platform and build breakdown:

• osx-shippable: 13
  • opt: 13
• osx-cross-devedition: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 13

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/54a722e614e6

Comment 14

[Iulian Moraru]

This is now permfailing as exception, the task that is failing now is BMN with the following failure line: signingscript.exceptions.SigningScriptError: Credentials not found for scope: project:releng:signing:cert:nightly-signing

• Push with failure
• Failure log

Comment 15

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Fix mac notarization scope r?#releng

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f7f8f094bd2d
Fix mac notarization scope r=releng-reviewers,gbrown

Comment 17

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/f7f8f094bd2d

Comment 18

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Add eme-free, l10n and partner repack notarization + pkg support r?#releng

Comment 19

[Intermittent Failures Robot]

14 failures in 3386 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• try: 9

Platform and build breakdown:

• osx-shippable: 10
  • opt: 10
• osx-cross-devedition: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-04-03&endday=2023-04-09&tree=all

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6eb460061cd1
Add eme-free, l10n and partner repack notarization + pkg support r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 21

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/6eb460061cd1

Comment 22

[Cosmin Sabou [:CosminS]]

Heitor, after https://hg.mozilla.org/mozilla-central/rev/6eb460061cd1 reached central these new tasks seem to retry a lot of times until after 5 retries they end as exception like here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=bmn&revision=1881ebd0d8e56fb3b338ca1eb047c6198b117e9a&group_state=expanded&selectedTaskRun=V4jDmNmmTYqZaJLTg6nJTw.5

Failure log: https://firefox-ci-tc.services.mozilla.com/tasks/V4jDmNmmTYqZaJLTg6nJTw/runs/7/logs/public/logs/live_backing.log

2023-04-15 16:48:26,059 - signingscript.utils - INFO - looking up notarization ticket for 2/2/b82fba3cb3d736fdf86304a614ac0be2b4496424
2023-04-15 16:48:26,205 - signingscript.utils - INFO - writing notarization ticket to /app/workdir/apple_notarize/Firefox Nightly.app/Contents/CodeResources
2023-04-15 16:48:26,207 - signingscript.utils - INFO - exitcode 0
2023-04-15 16:48:26,207 - signingscript.sign - INFO - Creating tarfile /app/workdir/public/build/nl/target.tar.gz...
2023-04-15 16:49:19,435 - signingscript.sign - DEBUG - _create_tarfile took 53.23s; RSS:181172 (+0)
2023-04-15 16:49:19,435 - signingscript.sign - DEBUG - apple_notarize took 163.43s; RSS:181172 (+0)
2023-04-15 16:49:19,436 - signingscript.utils - INFO - Copying /app/workdir/public/build/nl/target.tar.gz to /app/artifacts/public/build/nl/target.tar.gz
2023-04-15 16:49:19,577 - signingscript.utils - INFO - mkdir /app/workdir/public/build/nn-NO
2023-04-15 16:49:19,577 - signingscript.utils - INFO - Copying /app/workdir/cot/ZdsQ_F1TTpyVEc5ttDio0w/public/build/nn-NO/target.pkg to /app/workdir/public/build/nn-NO/target.pkg
2023-04-15 16:49:20,315 - signingscript.script - INFO - signing public/build/nn-NO/target.pkg
2023-04-15 16:49:20,315 - signingscript.task - INFO - sign(): Signing 156542460 bytes in /app/workdir/public/build/nn-NO/target.pkg with apple_notarization...
2023-04-15 16:49:20,529 - signingscript.utils - INFO - Running "rcodesign notary-submit --staple --api-key-path /app/configs/apple_api_key.json /app/workdir/apple_notarize/target.pkg"
2023-04-15 16:49:20,534 - signingscript.utils - INFO - COMMAND OUTPUT: 
2023-04-15 16:49:21,590 - signingscript.utils - INFO - creating Notary API submission for target.pkg (sha256: f268b57fd1cd1d720e45ad9d2bdadfe5efad1e5de9d9719783066687fec7fbc7)
2023-04-15 16:49:22,182 - signingscript.utils - INFO - created submission ID: bdcad070-0637-4437-b74e-8f7f98a6c502
2023-04-15 16:49:22,183 - signingscript.utils - INFO - resolving AWS S3 configuration from Apple-provided credentials
2023-04-15 16:49:22,185 - signingscript.utils - INFO - uploading asset to s3://notary-submissions-prod/prod/AROARQRX7CZS3PRF6ZA5L:bdcad070-0637-4437-b74e-8f7f98a6c502
2023-04-15 16:49:22,185 - signingscript.utils - INFO - (you may see additional log output from S3 client)
2023-04-15 16:49:24,261 - signingscript.utils - INFO - S3 upload completed successfully
2023-04-15 16:49:24,262 - signingscript.utils - INFO - waiting up to 600s for package upload bdcad070-0637-4437-b74e-8f7f98a6c502 to finish processing
2023-04-15 16:49:24,554 - signingscript.utils - INFO - poll state after 0s: InProgress
2023-04-15 16:49:27,766 - signingscript.utils - INFO - poll state after 3s: InProgress
2023-04-15 16:49:31,054 - signingscript.utils - INFO - poll state after 6s: InProgress
2023-04-15 16:49:34,392 - signingscript.utils - INFO - poll state after 10s: InProgress
2023-04-15 16:49:37,821 - signingscript.utils - INFO - poll state after 13s: InProgress
2023-04-15 16:49:41,065 - signingscript.utils - INFO - poll state after 16s: InProgress
2023-04-15 16:49:44,461 - signingscript.utils - INFO - poll state after 20s: InProgress
2023-04-15 16:49:47,808 - signingscript.utils - INFO - poll state after 23s: InProgress
2023-04-15 16:49:51,161 - signingscript.utils - INFO - poll state after 26s: InProgress
2023-04-15 16:49:54,740 - signingscript.utils - INFO - poll state after 30s: InProgress
2023-04-15 16:49:58,034 - signingscript.utils - INFO - poll state after 33s: InProgress
2023-04-15 16:50:01,333 - signingscript.utils - INFO - poll state after 37s: InProgress
2023-04-15 16:50:04,691 - signingscript.utils - INFO - poll state after 40s: InProgress
2023-04-15 16:50:08,132 - signingscript.utils - INFO - poll state after 43s: InProgress
Automation Error: python exited with signal -15

Could you please have a look over these? Thank you.

Comment 23

[Intermittent Failures Robot]

8 failures in 3206 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform and build breakdown:

• osx-shippable: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 24

[Julien Cristau [:jcristau]]

Looks like this is running into the 20min limit for scriptworker tasks (https://github.com/mozilla-releng/scriptworker-scripts/blob/2644bb85584b51f8647c7a749d920f35bd2c25d3/docker.d/pre-stop.sh#L19).
I guess we either need to bump that and terminationGracePeriodSeconds, run more chunks, or submit files to the notary service in parallel to cut down on the delays?

Comment 25

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1774273 - Fix notarization worker timeout (#710)

Comment 26

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Transition mac notarization to signingscript r?#taskgraph-reviewers,bhearsum,jcristau

Comment 27

[Intermittent Failures Robot]

20 failures in 650 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 18

Platform and build breakdown:

• osx-shippable: 19
  • opt: 19
• osx-cross-devedition: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-04-21&endday=2023-04-21&tree=all

Comment 28

[Intermittent Failures Robot]

39 failures in 4004 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 37
• mozilla-beta: 2

Platform and build breakdown:

• osx-shippable: 38
  • opt: 38
• osx-cross-devedition: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-04-17&endday=2023-04-23&tree=all

Comment 29

[Intermittent Failures Robot]

9 failures in 3177 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9

Platform and build breakdown:

• osx-shippable: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-04-24&endday=2023-04-30&tree=all

Comment 30

[Intermittent Failures Robot]

3 failures in 3411 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• osx-shippable: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1774273&startday=2023-05-01&endday=2023-05-07&tree=all

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4373378c3193
Transition mac notarization to signingscript r=taskgraph-reviewers,bhearsum

Comment 32

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4373378c3193

Comment 33

[Mike Hommey [:glandium]]

It looks like this broke decision tasks on the toolchains project branch:
https://treeherder.mozilla.org/logviewer?job_id=416923530&repo=toolchains&lineNumber=1479

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/6b82236cab6f
Followup. a=glandium

Comment 35

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6b82236cab6f

Comment 36

[Mike Hommey [:glandium]]

The followup was not enough :( https://treeherder.mozilla.org/logviewer?job_id=416933079&repo=toolchains&lineNumber=3285

Comment 37

[Julien Cristau [:jcristau]]

Attachment: Bug 1774273 - reverse logic for choosing mac-signing vs mac-notarization. r?hneiva

It seems safer to select signing by default, and notarization only on
select production projects, otherwise project branches such as toolchains
break because they end up requiring notarization jobs that are filtered
out by not being level 3.

Comment 38

[Julien Cristau [:jcristau]]

Now that the main piece has landed it's probably time to close this and handle followups in separate bugs, to make tracking easier.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dbf40fe5a4d5
reverse logic for choosing mac-signing vs mac-notarization. r=hneiva

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/e09ca28cc527
reverse logic for choosing mac-signing vs mac-notarization. r=hneiva

Comment 41

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/e09ca28cc527

Comment 42

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/dbf40fe5a4d5

Comment 43

[Heitor Neiva [:hneiva]]

Attachment: Bug 1774273 - Transition mac geckodriver notarization to signingscript r?#releng

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01dffee0963a
Transition mac geckodriver notarization to signingscript r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 45

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/01dffee0963a

Comment 47

[Ryan VanderMeulen [:RyanVM]]

Per the email Apple recently sent out, "Apple notary service will no longer accept uploads from altool as of November 1, 2023" so I think we're good for ESR102 as it'll be past EOL before then. Please confirm and set the status to wontfix if you agree, Heitor!

Comment 48

[Heitor Neiva [:hneiva]]

@RyanVM, agree that we likely won't need to uplift anything to esr102.
FWIW esr115 already has the new code and should be good.

For esr102 it's a "wontfix".