1860468 - Perma tier2 macosx1100-64-aarch64-qr TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nerli [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=433371853&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SMsbEeJzRz2n0J8duHKERg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-10-22T23:52:28.753Z] 23:52:28     INFO - runtests.py | Waiting for browser...
[task 2023-10-22T23:52:28.754Z] 23:52:28     INFO - GECKO(1476) | 1698018748754	Marionette	DEBUG	Closed connection 1
[task 2023-10-22T23:52:30.726Z] 23:52:30     INFO - GECKO(1476) | [Parent 1476, IndexedDB #8] WARNING: QM_TRY failure (WARNING): 'ToResult(!madeProgress) failed with resultCode 0x80004005, resultName NS_ERROR_FAILURE', file dom/indexedDB/ActorsParent.cpp:7020
[task 2023-10-22T23:52:30.727Z] 23:52:30     INFO - GECKO(1476) | [Parent 1476, IndexedDB #8] WARNING: QM_TRY failure (WARNING): 'ToResult(!madeProgress) failed with resultCode 0x80004005, resultName NS_ERROR_FAILURE', file dom/indexedDB/ActorsParent.cpp:7020
[task 2023-10-22T23:52:30.727Z] 23:52:30     INFO - GECKO(1476) | [Parent 1476, IndexedDB #8] WARNING: QM_TRY failure (WARNING): 'ToResult(!madeProgress) failed with resultCode 0x80004005, resultName NS_ERROR_FAILURE', file dom/indexedDB/ActorsParent.cpp:7020
[task 2023-10-22T23:52:30.728Z] 23:52:30     INFO - GECKO(1476) | [Parent 1476, IndexedDB #8] WARNING: QM_TRY failure (WARNING): 'ToResult(!madeProgress) failed with resultCode 0x80004005, resultName NS_ERROR_FAILURE', file dom/indexedDB/ActorsParent.cpp:7020
[task 2023-10-22T23:52:30.729Z] 23:52:30     INFO - GECKO(1476) | [Parent 1476, IndexedDB #8] WARNING: QM_TRY failure (WARNING): 'ToResult(!madeProgress) failed with resultCode 0x80004005, resultName NS_ERROR_FAILURE', file dom/indexedDB/ActorsParent.cpp:7020
[task 2023-10-22T23:58:40.791Z] 23:58:40     INFO - TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output
[task 2023-10-22T23:58:40.791Z] 23:58:40     INFO - TEST-INFO 
[task 2023-10-22T23:58:40.792Z] 23:58:40     INFO - Buffered messages finished
[task 2023-10-22T23:58:40.792Z] 23:58:40  WARNING - Force-terminating active process(es).
[task 2023-10-22T23:58:40.793Z] 23:58:40     INFO - Determining child pids from psutil...

Comment 1

[Noemi Erli[:noemi_erli]]

This is almost permafailing on central, since job macosx1100-64-aarch64-qr debug was added:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&collapsedPushes=842382&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=macosx1100-64-aarch64-qr%2Cdebug%2Cmochitests%2Ctest-macosx1100-64-aarch64-qr%2Fdebug-mochitest-media%2Cmda2&fromchange=7df8f9c41c9b60e87a707e41745479aaefaf782e&tochange=72b3ba30b666e344ae1d4220069e3659a3807530
Julien, could you please take a look at this?

Comment 2

[Intermittent Failures Robot]

4 failures in 3484 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 3

[Julien Cristau [:jcristau]]

[task 2023-10-17T22:18:05.702Z] 22:18:05 INFO - GECKO(4006) | dyld: Library not loaded: @rpath/libnss3.dylib
[task 2023-10-17T22:18:05.702Z] 22:18:05 INFO - GECKO(4006) | Referenced from: /opt/worker/tasks/task_169757806727819/build/application/Firefox NightlyDebug.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
[task 2023-10-17T22:18:05.702Z] 22:18:05 INFO - GECKO(4006) | Reason: no suitable image found. Did find:
[task 2023-10-17T22:18:05.702Z] 22:18:05 INFO - GECKO(4006) | /opt/worker/tasks/task_169757806727819/build/application/Firefox NightlyDebug.app/Contents/MacOS/plugin-container.app/Contents/MacOS/../../../libnss3.dylib: code signature in (/opt/worker/tasks/task_169757806727819/build/application/Firefox NightlyDebug.app/Contents/MacOS/plugin-container.app/Contents/MacOS/../../../libnss3.dylib) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)
[task 2023-10-17T22:18:05.702Z] 22:18:05 INFO - GECKO(4006) | /opt/worker/tasks/task_169757806727819/build/application/Firefox NightlyDebug.app/Contents/MacOS/plugin-container.app/Contents/MacOS/../../../libnss3.dylib: stat() failed with errno=1

Heitor, Haik, any idea what's going wrong with the nss signature here? I'm surprised that this is intermittent, could it be differences in test worker configs (I can see some pushes have a mix of green and busted tests)?

Comment 4

[Haik Aftandilian [:haik]]

More debugging needed to confirm, but the root cause of the problem appears to be that we are signing debug builds with our self-signed "Mozilla Fake DMG" cert while also using the production entitlement configuration. With the fix for bug 1593072, our production entitlement configuration is no longer compatible with self-signing. Until we have improved our codesigning to not-use self-signed certs, we'll have to either 1) sign our Debug central builds with the official cert OR 2) change our Debug central builds to use the development entitlement configuration.

If this evaluation is correct, this problem should not apply to try builds.

To explain this failure, plugin-container is configured to not allow loading of third party dylibs in production builds, but that setting is not compatible with self-signed certs. For our developer builds (which use self-signing), we opt out of that restriction. This incompatibility (where self-sign builds require the entitlement com.apple.security.cs.disable-library-validation in order to load a dylib that is also self-signed) must not apply on our macOS 10.15 x64 runners. One explanation could be that Apple Silicon or macOS 11 is more strict in this regard. The test is passing on macOS 10.15 on x64 and the failures are on macOS 11 on Apple Silicon. Apple Silicon does have more strict codesigning requirements per documentation from Apple, so it wouldn't be surprising if that was the difference.

macOS 11 (aka 10.16) is the first version to support Apple Silicon.

Regarding the solution, we do not sign non-official builds with production certs to avoid any potential confusion or malicious behavior where non-official build is passed off as official. For example a try build, but we might be OK with debug builds of central.

I'll discuss more with Heitor.

Comment 5

[Heitor Neiva [:hneiva]]

Attachment: Bug 1860468 - Use nonprod entitlements on non-shippable builds r?haik,#releng

Comment 6

[Intermittent Failures Robot]

17 failures in 638 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 7
• mozilla-central: 9
• autoland: 1

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 17
  • debug: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-10-27&endday=2023-10-27&tree=all

Comment 7

[Intermittent Failures Robot]

69 failures in 3657 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 30
• mozilla-beta: 38
• autoland: 1

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 68
  • debug: 68
• macosx1100-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 8

[Intermittent Failures Robot]

23 failures in 574 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 17
• mozilla-central: 6

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 22
  • debug: 22
• macosx1100-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-10-30&endday=2023-10-30&tree=all

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/65f4cd99ae3f
Use nonprod entitlements on non-shippable builds r=haik,releng-reviewers,taskgraph-reviewers,bhearsum,jcristau

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/65f4cd99ae3f

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/cf559e2b9cc6
Adjust Thunderbird mac-signing config. rs=bustage-fix

Comment 12

[Intermittent Failures Robot]

57 failures in 3356 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 38
• mozilla-central: 18
• try: 1

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 55
  • debug: 55
• macosx1100-64-shippable-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-10-30&endday=2023-11-05&tree=all

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Heitor, could this get an uplift rquest for beta, please?

Comment 14

[Heitor Neiva [:hneiva]]

Attachment: Bug 1860468 - Use nonprod entitlements on non-shippable builds

Removes the direct relationship between hardened-signing type (production/developer) and project/release level. Having them separate should make it more explicit on when to use production entitlements or not.

Original Revision: https://phabricator.services.mozilla.com/D191705

Comment 15

[Phabricator Automation]

Uplift Approval Request

• String changes made/needed: None
• Steps to reproduce for manual QE testing: Tests failing in CI without this patch
• Needs manual QE test: no
• Explanation of risk level: Only non-shippable builds were changed
• User impact if declined: None
• Fix verified in Nightly: yes
• Risk associated with taking this patch: No foreseable risk
• Is Android affected?: no
• Code covered by automated testing: no

Comment 16

[Dianna Smith [:diannaS]]

Comment on attachment 9362382 [details]
Bug 1860468 - Use nonprod entitlements on non-shippable builds

Approved for 120.0b8

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/997eb70f590c

Comment 18

[Magnus Melin [:mkmelin]]

https://bugzilla.mozilla.org/show_bug.cgi?id=1860468#c11 needs to go to comm-beta

Comment 19

[Rob Lemley [:rjl]]

Thunderbird 120.0b5:
https://hg.mozilla.org/releases/comm-beta/rev/68c447e6fb43

Comment 20

[Intermittent Failures Robot]

6 failures in 3288 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6

Platform and build breakdown:

• macosx1100-64-aarch64-qr: 6
  • debug: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1860468&startday=2023-11-06&endday=2023-11-12&tree=all