1853913 - Perma macOS 11 shippable toolkit/crashreporter/test/unit_ipc/test_content_phc3.js | xpcshell return code: 0

Status: REOPENED

(Toolkit :: Crash Reporting, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=429668109&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BA1rYtpSTJO1BXCM7KpoaA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-09-19T12:29:43.469Z] 12:29:43     INFO -  TEST-START | toolkit/crashreporter/test/unit_ipc/test_content_phc3.js
[task 2023-09-19T12:29:43.758Z] 12:29:43  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc3.js | xpcshell return code: 0
[task 2023-09-19T12:29:43.758Z] 12:29:43     INFO -  TEST-INFO took 289ms

Comment 1

[Cosmin Sabou [:CosminS]]

This started to fail with this merge https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=0a60f8be5517ddc928471d4a378b46228773fbc5
and fails only on mozilla-central.
Could be from https://hg.mozilla.org/mozilla-central/rev/8e3c8eba5d70b0f35e69af8b01b083ce4b75f246 or maybe https://hg.mozilla.org/mozilla-central/rev/330a48f1b6ad46d41bf24c85a6cc8aeb2d4621b5
Gabriele, could you have a look over it? Thank you.

Full list of test failing is this one:

7278	12:29:43 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc3.js | xpcshell return code: 0
7348	12:29:44 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc2.js | xpcshell return code: 0
7418	12:29:44 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc.js | xpcshell return code: 0
7548	12:29:44 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | xpcshell return code: 0
7630	12:29:45 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_large_annotation.js | xpcshell return code: 0
7700	12:29:45 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | xpcshell return code: 0
7770	12:29:45 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation.js | xpcshell return code: 0

Comment 2

[Intermittent Failures Robot]

30 failures in 3850 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #45 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 30

Platform and build breakdown:

• macosx1100-64-shippable-qr: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 3

[Gabriele Svelto [:gsvelto]]

I suspect this was caused by bug 1593072. This line in the log is the smoking gun:

INFO - PID 7676 | JavaScript error: resource://test/CrashTestUtils.sys.mjs, line 53: Error: couldn't open library /opt/worker/tasks/task_169512460876493/build/tests/xpcshell/tests/toolkit/crashreporter/test/libtestcrasher.dylib: dlopen(/opt/worker/tasks/task_169512460876493/build/tests/xpcshell/tests/toolkit/crashreporter/test/libtestcrasher.dylib, 2): no suitable image found. Did find:

Haik, we build a special library to crash the xpcshell tests, it's call testcrasher and built here. Do I need to change it in some way to accommodate for the new entitlements?

Comment 4

[Gabriele Svelto [:gsvelto]]

Forgot the NI? see my question in comment 3.

Comment 5

[Haik Aftandilian [:haik]]

(In reply to Gabriele Svelto [:gsvelto] from comment #3)

I suspect this was caused by bug 1593072. This line in the log is the smoking gun:

INFO - PID 7676 | JavaScript error: resource://test/CrashTestUtils.sys.mjs, line 53: Error: couldn't open library /opt/worker/tasks/task_169512460876493/build/tests/xpcshell/tests/toolkit/crashreporter/test/libtestcrasher.dylib: dlopen(/opt/worker/tasks/task_169512460876493/build/tests/xpcshell/tests/toolkit/crashreporter/test/libtestcrasher.dylib, 2): no suitable image found. Did find:

Haik, we build a special library to crash the xpcshell tests, it's call testcrasher and built here. Do I need to change it in some way to accommodate for the new entitlements?

Thanks for the background. This is definitely looks like a regression from the entitlement fixes and the fix is probably going to be to change how the executable doing the loading is signed.

With bug 1593072, most of our executables are not permitted to load third party libraries, but that restriction is not compatible with how we self-sign non-production try builds so we relax it on those builds. That is not happening here so we need to fix that. I'll take the bug and look into this some more.

In the full log, this part gives it a away not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?).

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1593072

Comment 7

[Haik Aftandilian [:haik]]

The problem here is that when the test is run on the official mozilla-central repo, it is run with our production cert and entitlements which prevent the parent process firefox executable and plugin-container executables from loading dylibs not signed by Mozilla or Apple.

The xpcshell executable being used by the test is not signed and therefore not subject to the entitlements, but xpcshell spawns a plugin-container executable which loads libtestcrasher.dylib and since libtestcrasher.dylib is not signed, the load is not permitted by macOS.

This test should pass for try pushes because those builds use our developer entitlements which permit loading of third party libraries by plugin-container to workaround other issues with signing.

Solutions I can think of (from least to most hacky):

1. :gsvelto mentioned that this test could be refactored to not use libtestcrasher.dylib. That seems like the best approach.

2. Have our signing infrastructure sign libtestcrasher.dylib which would allow it to be loaded by plugin-container on production builds.

3. Tests that need to inject a dylib on production builds could first strip off the signature and entitlements from executables so that injection can work. (This can be done with the codesign command.) The downside of this is that the tests would not be as representative of real world use because the entitlements might have other impacts affecting test execution.

I'll look into these options.

Sorry we didn't catch this before landing bug 1593072. We tested with production certs, but didn't run this test.

Comment 8

[Intermittent Failures Robot]

54 failures in 3726 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 36
• mozilla-beta: 18

Platform and build breakdown:

• macosx1100-64-shippable-qr: 54
  • opt: 54

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 9

[Intermittent Failures Robot]

55 failures in 4018 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 14
• mozilla-central: 41

Platform and build breakdown:

• macosx1100-64-shippable-qr: 55
  • opt: 55

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-02&endday=2023-10-08&tree=all

Comment 10

[Cosmin Sabou [:CosminS]]

Haik, any updates on this one? Being that it's permanently crashing should we consider disabling these tests on macOS 11?

Comment 11

[Haik Aftandilian [:haik]]

@Cosmin, with the workaround fix just landed for bug 1856972, the test should stop failing. Eventually we'll have to revisit this problem and update the test. Once it is confirmed that the test is passing again, we can close this bug.

Comment 12

[Cosmin Sabou [:CosminS]]

Still failing after those changes reached central: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=os%2Cx%2C11%2Cship%2Cxpc&revision=6769216667ff487b1442f654ee41dc5f12c23983&selectedTaskRun=APaw57pLTzaeIm8LBedSOg.0
List of tests:

7912	06:47:18 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_oom_annotation.js | xpcshell return code: 0
7975	06:47:18 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crashreporter_crash.js | xpcshell return code: 0
8033	06:47:18 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_annotation.js | xpcshell return code: 0
8108	06:47:18 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_large_annotation.js | xpcshell return code: 0
8171	06:47:19 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_exception_time_annotation.js | xpcshell return code: 0
8234	06:47:19 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_oom_annotation.js | xpcshell return code: 0


6233	06:46:28 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc.js | xpcshell return code: 0
6296	06:46:28 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc2.js | xpcshell return code: 0
6359	06:46:28 WARNING - TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit_ipc/test_content_phc3.js | xpcshell return code: 0

Comment 13

[Haik Aftandilian [:haik]]

(In reply to Haik Aftandilian [:haik] from comment #11)

@Cosmin, with the workaround fix just landed for bug 1856972, the test should stop failing. Eventually we'll have to revisit this problem and update the test. Once it is confirmed that the test is passing again, we can close this bug.

Ah, thanks. My comment was incorrect. The fix for bug 1856972 affected the parent process executable only and this test is failing because of the plugin-container child process entitlements (per comment 7). I would rather not disable the test because it should still be passing for try pushes. If we could disable it only for mozilla-central pushes, that could be a good short term workaround. I will try to get to this next week.

Comment 15

[Cosmin Sabou [:CosminS]]

You're right, as it does not affect try pushes will leave it as is until a fix is in place. Thanks for looking into it.

Comment 17

[Intermittent Failures Robot]

18 failures in 638 pushes (0.028 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 14
• mozilla-beta: 4

Platform and build breakdown:

• macosx1100-64-shippable-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-13&endday=2023-10-13&tree=all

Comment 18

[Intermittent Failures Robot]

85 failures in 3764 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 30
• mozilla-central: 55

Platform and build breakdown:

• macosx1100-64-shippable-qr: 85
  • opt: 85

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 19

[Intermittent Failures Robot]

51 failures in 3484 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 5
• mozilla-central: 44
• mozilla-release: 2

Platform and build breakdown:

• macosx1100-64-shippable-qr: 51
  • opt: 51

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-16&endday=2023-10-22&tree=all

Comment 20

[Haik Aftandilian [:haik]]

I had thought it might be reasonable to add a workaround for this particular test involving stripping the signatures from our executables and re-signing in a way that allows loading of libtestcrasher.dylib. But after looking at xpcshell tests more, I don't see a way to do that without in a minimal way.

@gsvelto, you had mentioned possibly rewriting this test to not depend on loading a dylib. Did you have a method in mind? It sounds like that would mean adding the crashing code in shipping builds.

Alternatively, we could have libtestcrasher.dylib be signed. @Heitor, could you comment on how complex that would be?

Comment 21

[Gabriele Svelto [:gsvelto]]

I would stuff the various crash methods into nsIDebug2, we already have some there like this one. This would make the crash methods available to mochitests too, which would be nice.

Comment 22

[Intermittent Failures Robot]

55 failures in 3657 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 16
• mozilla-central: 38
• mozilla-release: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 55
  • opt: 55

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 23

[Intermittent Failures Robot]

17 failures in 574 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 8
• mozilla-beta: 7
• mozilla-release: 2

Platform and build breakdown:

• macosx1100-64-shippable-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-30&endday=2023-10-30&tree=all

Comment 24

[Natalia Csoregi [:nataliaCs]]

Update

There have been 40 total failures within the last 7 days, all of them on OS X 11 WebRender Shippable opt.

Recent log: https://treeherder.mozilla.org/logviewer?job_id=435024551&repo=mozilla-central&lineNumber=9073

Comment 25

[Intermittent Failures Robot]

17 failures in 216 pushes (0.079 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 8
• try: 2
• mozilla-release: 5

Platform and build breakdown:

• macosx1100-64-shippable-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-04&endday=2023-11-04&tree=all

Comment 26

[Intermittent Failures Robot]

77 failures in 3356 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 43
• mozilla-beta: 23
• mozilla-release: 9
• try: 2

Platform and build breakdown:

• macosx1100-64-shippable-qr: 77
  • opt: 77

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-10-30&endday=2023-11-05&tree=all

Comment 27

[Heitor Neiva [:hneiva]]

Attachment: WIP: - Bug 1853913 - Sign libtestcrasher.dylib for mac tests

Comment 28

[Heitor Neiva [:hneiva]]

I was able to pull libtestcrasher.dylib from the build and sign it in a separate task. I poked around for awhile at the test tasks but was unable to (A) find where the signed file needs to be fetched and (B) how to unpack and use it.

Anyone with more experience than me getting around the test tasks ?

Comment 29

[Intermittent Failures Robot]

22 failures in 658 pushes (0.033 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 16
• mozilla-central: 6

Platform and build breakdown:

• macosx1100-64-shippable-qr: 22
  • opt: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-09&endday=2023-11-09&tree=all

Comment 30

[Intermittent Failures Robot]

70 failures in 3288 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 39
• mozilla-central: 25
• mozilla-release: 6

Platform and build breakdown:

• macosx1100-64-shippable-qr: 70
  • opt: 70

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-06&endday=2023-11-12&tree=all

Comment 31

[Intermittent Failures Robot]

48 failures in 3495 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 39
• mozilla-beta: 3
• mozilla-release: 6

Platform and build breakdown:

• macosx1100-64-shippable-qr: 48
  • opt: 48

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-13&endday=2023-11-19&tree=all

Comment 32

[Natalia Csoregi [:nataliaCs]]

Update

There have been 48 total failures within the last 7 days, all of them on OS X 11 WebRender Shippable opt.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=437641779&repo=mozilla-central&lineNumber=5276

Comment 33

[Intermittent Failures Robot]

69 failures in 3623 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #21 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 45
• mozilla-beta: 24

Platform and build breakdown:

• macosx1100-64-shippable-qr: 69
  • opt: 69

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 34

[Intermittent Failures Robot]

17 failures in 700 pushes (0.024 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 3
• mozilla-central: 10
• mozilla-release: 4

Platform and build breakdown:

• macosx1100-64-shippable-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-27&endday=2023-11-27&tree=all

Comment 35

[Intermittent Failures Robot]

17 failures in 821 pushes (0.021 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 9
• mozilla-beta: 7
• mozilla-release: 1

Platform and build breakdown:

• macosx1100-64-shippable-qr: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-30&endday=2023-11-30&tree=all

Comment 36

[Intermittent Failures Robot]

79 failures in 4108 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #17 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 26
• mozilla-central: 36
• mozilla-release: 17

Platform and build breakdown:

• macosx1100-64-shippable-qr: 79
  • opt: 79

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 37

[Intermittent Failures Robot]

43 failures in 3671 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 17
• mozilla-beta: 23
• mozilla-release: 3

Platform and build breakdown:

• macosx1100-64-shippable-qr: 43
  • opt: 43

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 38

[Narcis Beleuzu [:NarcisB]]

In the last 2 weeks there is no occurrency of the fail

Comment 39

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 40

[Haik Aftandilian [:haik]]

needinfo for myself to resolve this so the test can continue to be run.

Comment 41

[Intermittent Failures Robot]

4 failures in 4547 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• windows11-32-2009-qr: 2
  • opt: 2
• windows11-64-2009-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2024-05-13&endday=2024-05-19&tree=all

Comment 42

[Intermittent Failures Robot]

Summary

1 failures in 4309 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

Table

	spi-nw
macosx1015-64-qr--/debug	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1853913&startday=2025-02-03&endday=2025-02-09&tree=all