1873329 - Perma geckodriver-mac-notarization fails as exception with Error: appstore connect error | This request requires an in-effect agreement that has not been signed or has expired

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[Iulian Moraru]

The geckodriver-mac-notarization jobs are permafailing as exceptions -> OS X opt and OS X AArch64 opt

• Push where this first happened
• Failure log
• Task link for OS X AArch64 opt
• Task link for OS X opt

2024-01-06 09:48:42,308 - signingscript.sign - DEBUG - _create_zipfile took 0.33s; RSS:348268 (+0)
2024-01-06 09:48:42,309 - signingscript.rcodesign - INFO - Running command: rcodesign notary-submit --api-key-path /app/configs/apple_api_key.json /app/workdir/apple_notarize/geckodriver.zip
2024-01-06 09:48:42,315 - signingscript.rcodesign - INFO - COMMAND OUTPUT: 
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING - creating Notary API submission for geckodriver.zip (sha256: 99c9d2c913c542ddf18eb486fb52b5331f9c6237a1f8731392658618a8ccf51e)
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING - Error: appstore connect error:
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING - POST https://appstoreconnect.apple.com/notary/v2/submissions
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING - {
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -   "errors": [
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -     {
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -       "code": "FORBIDDEN.REQUIRED_AGREEMENTS_MISSING_OR_EXPIRED",
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -       "detail": "This request requires an in-effect agreement that has not been signed or has expired.",
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -       "id": "ESUTH4HJKNY3Q3X637HBOR537M",
2024-01-06 09:48:42,770 - signingscript.rcodesign - WARNING -       "links": {
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -         "see": "/agreements"
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -       },
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -       "status": "403",
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -       "title": "A required agreement is missing or has expired."
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -     }
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING -   ]
2024-01-06 09:48:42,771 - signingscript.rcodesign - WARNING - }
2024-01-06 09:48:42,771 - signingscript.rcodesign - INFO - exitcode 1
2024-01-06 09:48:42,771 - scriptworker.utils - DEBUG - retry_async: rcodesign_notarize: sleeping 13.574302702374808 seconds before retry

Comment 1

[Iulian Moraru]

Hi Johan! Can you please take a look at this?
Thank you!

Comment 2

[Iulian Moraru]

Later, this 2 build-mac-notarization jobs failed on mozilla-beta:

• OS X Cross Compiled Shippable
• OS X Cross Compiled DevEdition opt

Comment 3

[Haik Aftandilian [:haik]]

I suspect we need to agree to the latest terms and conditions on our Apple accounts. On our Apple Developer ID account, I see the below message when logging in. One of our Releng account owners should be able to take care of this.

The program license agreement has been updated.

Mozilla Release Engineering needs to accept the latest Apple Developer
Program License Agreement for your team to continue integrating new
technologies, submitting content to App Review, accessing Certificates,
Identifiers and Profiles, and more.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

This'll block tomorrow's planned dot release and any other planned release activity. Severity->Blocker.

Comment 5

[Donal Meehan [:dmeehan]]

This also blocks uploading iOS apps to the App Store. Until the updated Apple Developer Program Licence Agreement is accepted, we can't submit new iOS builds.

Example from the 122 beta build:
https://app.bitrise.io/build/15892d35-fd84-4928-b69e-6a27e8a55d47

You do not have required contracts to perform an operation

Comment 6

[Intermittent Failures Robot]

37 failures in 2274 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 19
• mozilla-beta: 18

Platform and build breakdown:

• macosx64-aarch64: 8
  • opt: 8
• macosx64: 7
  • opt: 7
• osx-shippable: 11
  • opt: 11
• osx-cross-devedition: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873329&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 7

[Johan Lorenzo [:jlorenzo]]

We were warned by Apple a few days ago. :hneiva immediately filed a CASA ticket[1] to get Legal's approval. I'll escalate this to them.

[1] https://biztera.com/projects/19638

Comment 9

[Heitor Neiva [:hneiva]]

The agreement has been accepted.

Comment 10

[Intermittent Failures Robot]

10 failures in 3217 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• mozilla-beta: 3
• mozilla-release: 2

Platform and build breakdown:

• macosx64: 2
  • opt: 2
• macosx64-aarch64: 2
  • opt: 2
• osx-shippable: 4
  • opt: 4
• osx-cross-devedition: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873329&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 11

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: Bug 1873329 - Add a shutdown blocker for NotificationDB r=asuth,jstutte

Comment 12

[Phabricator Automation]

Comment on attachment 9377227 [details]
Bug 1873329 - Add a shutdown blocker for NotificationDB r=asuth,jstutte

Revision D200042 was moved to bug 1873229. Setting attachment 9377227 [details] to obsolete.