1673094 - Intermittent iscript.exceptions.IScriptError: ('%s in %s exited %s!\n%s', ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-<mac-release|mac-nightly>.1603480938.134891',

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=319582414&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GfzW9EcjS82crrESTUMV8A/runs/0/artifacts/public/logs/live_backing.log

---

2020-10-23 12:22:17,986 - scriptworker_client.utils - INFO -   adding: 0/Firefox.pkg (deflated 0%)
2020-10-23 12:22:18,134 - scriptworker_client.utils - INFO - ['zip', '-r', '/builds/scriptworker/work/notarization.zip', '0/Firefox.app', '0/Firefox.pkg'] in /builds/scriptworker/work exited 0
2020-10-23 12:22:18,135 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2020-10-23 12:22:25,809 - scriptworker_client.utils - INFO - 2020-10-23 12:22:25.809 altool[30085:86310977] *** Error: Unable to validate your application. We are unable to create an authentication session.
2020-10-23 12:22:25,853 - scriptworker_client.aio - DEBUG - retry_async: run_command: sleeping 12.135000474570731 seconds before retry
2020-10-23 12:22:38,139 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2020-10-23 12:22:39,966 - scriptworker_client.utils - INFO - 2020-10-23 12:22:39.966 altool[30136:86311348] *** Error: Unable to validate your application. We are unable to create an authentication session.
2020-10-23 12:22:39,968 - scriptworker_client.aio - DEBUG - retry_async: run_command: sleeping 27.276434272722916 seconds before retry
2020-10-23 12:23:07,384 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2020-10-23 12:23:10,416 - scriptworker_client.utils - INFO - 2020-10-23 12:23:10.416 altool[30220:86311886] *** Error: Unable to validate your application. We are unable to create an authentication session.
2020-10-23 12:23:10,459 - scriptworker_client.aio - DEBUG - retry_async: run_command: sleeping 49.35676011603144 seconds before retry
2020-10-23 12:23:59,937 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2020-10-23 12:24:05,408 - scriptworker_client.utils - INFO - 2020-10-23 12:24:05.408 altool[30385:86312850] *** Error: Unable to validate your application. We are unable to create an authentication session.
2020-10-23 12:24:05,411 - scriptworker_client.aio - DEBUG - retry_async: run_command: sleeping 84.73006157268217 seconds before retry
2020-10-23 12:25:30,298 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2020-10-23 12:25:35,190 - scriptworker_client.utils - INFO - 2020-10-23 12:25:35.189 altool[30632:86314481] *** Error: Unable to validate your application. We are unable to create an authentication session.
2020-10-23 12:25:35,232 - scriptworker_client.aio - WARNING - retry_async: run_command: too many retries!
2020-10-23 12:25:35,232 - scriptworker_client.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/builds/scriptworker/virtualenv/src/scriptworker-client/scriptworker_client/src/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
    await async_main(config, task)
  File "/builds/scriptworker/virtualenv/src/iscript/iscript/src/iscript/script.py", line 38, in async_main
    await notarize_1_behavior(config, task)
  File "/builds/scriptworker/virtualenv/src/iscript/iscript/src/iscript/mac.py", line 1118, in notarize_1_behavior
    poll_uuids = await notarize_no_sudo(work_dir, key_config, zip_path)
  File "/builds/scriptworker/virtualenv/src/iscript/iscript/src/iscript/mac.py", line 750, in notarize_no_sudo
    await retry_async(
  File "/builds/scriptworker/virtualenv/src/scriptworker-client/scriptworker_client/src/scriptworker_client/aio.py", line 322, in retry_async
    return await func(*args, **kwargs)
  File "/builds/scriptworker/virtualenv/src/scriptworker-client/scriptworker_client/src/scriptworker_client/utils.py", line 232, in run_command
    raise exception(
iscript.exceptions.IScriptError: ('%s in %s exited %s!\n%s', ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1603480938.134891', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'], '/', 222, '')
exit code: 1```

Comment 1

[Intermittent Failures Robot]

2 failures in 4230 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2

Platform breakdown:

• osx-shippable: 1
• osx-cross-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673094&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 2

[Intermittent Failures Robot]

5 failures in 4275 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform breakdown:

• osx-shippable: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1673094&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 3683 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 4

[Intermittent Failures Robot]

4 failures in 4257 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 4

Platform and build breakdown:

• osx-shippable: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 5

[Intermittent Failures Robot]

11 failures in 3125 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• mozilla-central: 7

Platform and build breakdown:

• osx-cross-devedition: 4
  • opt: 4
• osx-shippable: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 6

[Intermittent Failures Robot]

16 failures in 3854 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• mozilla-beta: 8

Platform and build breakdown:

• osx-shippable: 13
  • opt: 13
• osx-cross-devedition: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 7

[Intermittent Failures Robot]

9 failures in 4255 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• mozilla-release: 3

Platform and build breakdown:

• osx-shippable: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 3224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• osx-cross-devedition: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 9

[Sandor Molnar[:smolnar]]

This has turned perma since the last merge on central:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&fromchange=f3aa64653924b1be713882b95438b29a4724ce39&searchStr=os%2Cx%2Ccross%2Ccompiled%2Cshippable%2Copt%2Cbuild-notarization-part-1-macosx64-shippable%2Fopt%2Cbn

Failure log

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Sandor Molnar from comment #9)

This has turned perma since the last merge on central:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&fromchange=f3aa64653924b1be713882b95438b29a4724ce39&searchStr=os%2Cx%2Ccross%2Ccompiled%2Cshippable%2Copt%2Cbuild-notarization-part-1-macosx64-shippable%2Fopt%2Cbn

Failure log

this is different than the original issue, but it's fallout from something i did today. working on a fix

Comment 11

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #10)

(In reply to Sandor Molnar from comment #9)

This has turned perma since the last merge on central:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&fromchange=f3aa64653924b1be713882b95438b29a4724ce39&searchStr=os%2Cx%2Ccross%2Ccompiled%2Cshippable%2Copt%2Cbuild-notarization-part-1-macosx64-shippable%2Fopt%2Cbn

Failure log

this is different than the original issue, but it's fallout from something i did today. working on a fix

This is fixed now. Rerunning any busted jobs should succeed.

Comment 12

[Sandor Molnar[:smolnar]]

(In reply to [e-mail] (:bhearsum) from comment #11)

(In reply to [e-mail] (:bhearsum) from comment #10)

(In reply to Sandor Molnar from comment #9)

This has turned perma since the last merge on central:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&fromchange=f3aa64653924b1be713882b95438b29a4724ce39&searchStr=os%2Cx%2Ccross%2Ccompiled%2Cshippable%2Copt%2Cbuild-notarization-part-1-macosx64-shippable%2Fopt%2Cbn

Failure log

this is different than the original issue, but it's fallout from something i did today. working on a fix

This is fixed now. Rerunning any busted jobs should succeed.

I can confirm. Jobs are green now.
I will change the summary of the bug back to intermittent to track the original issue.

Comment 13

[Intermittent Failures Robot]

10 failures in 3065 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• comm-central: 1
• mozilla-beta: 5

Platform and build breakdown:

• osx-shippable: 8
  • opt: 8
• osx-cross-devedition: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 14

[Intermittent Failures Robot]

10 failures in 3115 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-release: 1
• mozilla-beta: 6
• mozilla-esr78: 1

Platform and build breakdown:

• osx-shippable: 7
  • opt: 7
• osx-cross-devedition: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2021-06-21&endday=2021-06-27&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 3060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 3444 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2022-01-17&endday=2022-01-23&tree=all

Comment 17

[Julien Cristau [:jcristau]]

Recent failure is not related to the initial one, and looks like:

2022-01-19 05:40:42,043 - scriptworker_client.utils - INFO - Running ['xcrun', 'altool', '--notarize-app', '-f', '/builds/scriptworker/work/notarization.zip', '--primary-bundle-id', 'org.mozilla.firefox-mac-release.1642570842.41925', '-u', '[e-mail]', '--asc-provider', 'MozillaCorporation', '--password', '********'] in / ...
2022-01-19 05:41:12,224 - scriptworker_client.utils - INFO - 2022-01-19 05:41:12.224 altool[7224:56913430] *** Error: Server returned an invalid MIME type: text/html, body:
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - <!DOCTYPE html>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - <html lang="en">
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - <head>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -     <style>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -         body {
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             font-family: "Helvetica Neue", "HelveticaNeue", Helvetica, Arial, sans-serif;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             font-size: 15px;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             font-weight: 200;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             line-height: 20px;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             color: #4c4c4c;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             text-align: center;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -         }
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - 
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -         .section {
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -             margin-top: 50px;
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -         }
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -     </style>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - </head>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - <body>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - <div class="section">
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -     <h1>&#63743;</h1>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - 
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -     <h3>Bad Gateway</h3>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO -     <p>Correlation Key: 3MJYHICGO6G4B55IBPK3RN6WDA</p>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - </div>
2022-01-19 05:41:12,225 - scriptworker_client.utils - INFO - </body>
2022-01-19 05:41:12,226 - scriptworker_client.utils - INFO - </html>

which seems like an apple infra issue

Comment 18

[Intermittent Failures Robot]

19 failures in 707 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 13
• mozilla-central: 6

Platform and build breakdown:

• osx-shippable: 2
  • opt: 2
• osx-cross-devedition: 11
  • opt: 11
• macosx64: 5
  • opt: 5
• macosx64-aarch64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2023-02-17&endday=2023-02-17&tree=all

Comment 19

[Intermittent Failures Robot]

19 failures in 3856 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 13
• mozilla-central: 6

Platform and build breakdown:

• osx-shippable: 2
  • opt: 2
• osx-cross-devedition: 11
  • opt: 11
• macosx64: 5
  • opt: 5
• macosx64-aarch64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1673094&startday=2023-02-13&endday=2023-02-19&tree=all

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.