1969102 - Support for shared per-task setup and teardown in xpcshell tests

Status: ASSIGNED

(Testing :: General, enhancement)

Description

[Beth Rennie [:beth] (she/her)]

Bug 1957513 added support for all variants of mochitest, but xpcshell support did not land due to confusing test failures.

Comment 1

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1969102 - Add Mochia to xpcshell tests r?standard8!,gijs!

Comment 2

[Beth Rennie [:beth] (she/her)]

To try to understand why some tests are failing with these patches I kicked off a number of try jobs:

• Adding Mochia to head.js but not adding any methods:
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135274

• Adding only describe():
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135275

• Adding only beforeEach():
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135276

• Adding only afterEach():
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135277

• Adding only it():
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135278

Hopefully we will see some differences between these jobs and we can tease apart what exactly is happening.

ni? me to look at these results and comment here.

Comment 3

[Beth Rennie [:beth] (she/her)]

Another try job:

• Using loadSubScript but not calling Mochia():
  https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=135336

Comment 4

[Beth Rennie [:beth] (she/her)]

Hey :kershaw, I'm looking at the xpcshell failures from comment 2 and there almost entirely failing in X-spi-nw suites with some very curious error messages, e.g.

parsed log:

TEST-UNEXPECTED-FAIL | extensions/pref/autoconfig/test/unit/test_autoconfig.js | xpcshell return code: 0
[task 2025-06-04T19:47:00.050+00:00] 19:47:00     INFO -  TEST-INFO took 1575ms
[task 2025-06-04T19:47:00.050+00:00] 19:47:00     INFO -  >>>>>>>
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefService.resetPrefs]
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  run_test@/builds/worker/workspace/build/tests/xpcshell/tests/extensions/pref/autoconfig/test/unit/test_autoconfig.js:81:20
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:590:7
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  @-e:1:1
[task 2025-06-04T19:47:00.051+00:00] 19:47:00     INFO -  <<<<<<<
[task 2025-06-04T19:47:00.052+00:00] 19:47:00     INFO -  TEST-START | js/xpconnect/tests/unit/test_params.js
[task 2025-06-04T19:47:08.402+00:00] 19:47:08     INFO -  TEST-PASS | js/xpconnect/tests/unit/test_params.js | took 8349ms
[task 2025-06-04T19:47:08.404+00:00] 19:47:08     INFO -  TEST-START | extensions/pref/autoconfig/test/unit/test_autoconfig_no_sandbox.js
[task 2025-06-04T19:47:09.941+00:00] 19:47:09  WARNING -  TEST-UNEXPECTED-FAIL | extensions/pref/autoconfig/test/unit/test_autoconfig_no_sandbox.js | xpcshell return code: 0
[task 2025-06-04T19:47:09.941+00:00] 19:47:09     INFO -  TEST-INFO took 1536ms
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  >>>>>>>
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefService.resetPrefs]
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  run_test@/builds/worker/workspace/build/tests/xpcshell/tests/extensions/pref/autoconfig/test/unit/test_autoconfig_no_sandbox.js:57:20
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:590:7
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  @-e:1:1
[task 2025-06-04T19:47:09.942+00:00] 19:47:09     INFO -  <<<<<<<
[task 2025-06-04T19:47:09.943+00:00] 19:47:09     INFO -  TEST-START | extensions/pref/autoconfig/test/unit/test_autoconfig_nonascii.js
[task 2025-06-04T19:47:11.498+00:00] 19:47:11  WARNING -  TEST-UNEXPECTED-FAIL | extensions/pref/autoconfig/test/unit/test_autoconfig_nonascii.js | xpcshell return code: 0
[task 2025-06-04T19:47:11.498+00:00] 19:47:11     INFO -  TEST-INFO took 1554ms
[task 2025-06-04T19:47:11.498+00:00] 19:47:11     INFO -  >>>>>>>
[task 2025-06-04T19:47:11.499+00:00] 19:47:11     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2025-06-04T19:47:11.499+00:00] 19:47:11     INFO -  NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefService.resetPrefs]
[task 2025-06-04T19:47:11.499+00:00] 19:47:11     INFO -  run_test@/builds/worker/workspace/build/tests/xpcshell/tests/extensions/pref/autoconfig/test/unit/test_autoconfig_nonascii.js:108:20
[task 2025-06-04T19:47:11.499+00:00] 19:47:11     INFO -  _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:590:7
[task 2025-06-04T19:47:11.499+00:00] 19:47:11     INFO -  @-e:1:1
[task 2025-06-04T19:47:11.500+00:00] 19:47:11     INFO -  <<<<<<<

parsed log:

[task 2025-06-04T20:07:23.040+00:00] 20:07:23  WARNING -  TEST-UNEXPECTED-FAIL | modules/libpref/test/unit/test_locked_file_prefs.js | xpcshell return code: 0
[task 2025-06-04T20:07:23.042+00:00] 20:07:23     INFO -  TEST-INFO took 183ms
[task 2025-06-04T20:07:23.042+00:00] 20:07:23     INFO -  >>>>>>>
[task 2025-06-04T20:07:23.042+00:00] 20:07:23     INFO -  PID 5828 | JavaScript error: D:/task_174906597843341/build/tests/xpcshell/tests/modules/libpref/test/unit/head_libPrefs.js, line 28: NS_ERROR_FAILURE: Tried to get test directory 'ProfLDS'
[task 2025-06-04T20:07:23.042+00:00] 20:07:23     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2025-06-04T20:07:23.043+00:00] 20:07:23     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2025-06-04T20:07:23.043+00:00] 20:07:23     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2025-06-04T20:07:23.043+00:00] 20:07:23     INFO -  running event loop
[task 2025-06-04T20:07:23.043+00:00] 20:07:23     INFO -  "CONSOLE_MESSAGE: (error) [JavaScript Error: "NS_ERROR_FAILURE: Tried to get test directory 'ProfLDS'" {file: "D:/task_174906597843341/build/tests/xpcshell/tests/modules/libpref/test/unit/head_libPrefs.js" line: 28}]
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  getFile@D:/task_174906597843341/build/tests/xpcshell/tests/modules/libpref/test/unit/head_libPrefs.js:28:22
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  _execute_test@D:\\task_174906597843341\\build\\tests\\xpcshell\\head.js:576:48
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  @-e:1:1
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  "
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  modules/libpref/test/unit/test_locked_file_prefs.js | Starting notChangedFromAPI
[task 2025-06-04T20:07:23.044+00:00] 20:07:23     INFO -  (xpcshell/head.js) | test notChangedFromAPI pending (2)
[task 2025-06-04T20:07:23.045+00:00] 20:07:23     INFO -  NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefService.resetPrefs]
[task 2025-06-04T20:07:23.045+00:00] 20:07:23     INFO -  notChangedFromAPI@D:/task_174906597843341/build/tests/xpcshell/tests/modules/libpref/test/unit/test_locked_file_prefs.js:18:6
[task 2025-06-04T20:07:23.045+00:00] 20:07:23     INFO -  _run_next_test@D:\task_174906597843341\build\tests\xpcshell\head.js:1813:11
[task 2025-06-04T20:07:23.046+00:00] 20:07:23     INFO -  run@D:\task_174906597843341\build\tests\xpcshell\head.js:808:9
[task 2025-06-04T20:07:23.046+00:00] 20:07:23     INFO -  _do_main@D:\task_174906597843341\build\tests\xpcshell\head.js:245:6
[task 2025-06-04T20:07:23.046+00:00] 20:07:23     INFO -  _execute_test@D:\task_174906597843341\build\tests\xpcshell\head.js:596:5
[task 2025-06-04T20:07:23.046+00:00] 20:07:23     INFO -  @-e:1:1
[task 2025-06-04T20:07:23.046+00:00] 20:07:23     INFO -  exiting test
[task 2025-06-04T20:07:23.047+00:00] 20:07:23     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2025-06-04T20:07:23.047+00:00] 20:07:23     INFO -  "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
[task 2025-06-04T20:07:23.047+00:00] 20:07:23     INFO -  PID 5828 | JavaScript error: D:\\task_174906597843341\\build\\tests\\xpcshell\\head.js, line 738: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIXPCComponents_Utils.exitIfInAutomation]
[task 2025-06-04T20:07:23.047+00:00] 20:07:23     INFO -  <<<<<<<

and this log:

[task 2025-06-04T20:10:47.932+00:00] 20:10:47     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2025-06-04T20:10:47.932+00:00] 20:10:47     INFO -  Unexpected exception NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPrefService.resetPrefs]
[task 2025-06-04T20:10:47.933+00:00] 20:10:47     INFO -  migrateSanitizationPrefsClearCleaningPrefs@D:/task_174906583785244/build/tests/xpcshell/tests/netwerk/cookie/test/unit/test_migrateCookieLifetimePref.js:91:18
[task 2025-06-04T20:10:47.933+00:00] 20:10:47     INFO -  _run_next_test/<@D:\task_174906583785244\build\tests\xpcshell\head.js:1759:22
[task 2025-06-04T20:10:47.933+00:00] 20:10:47     INFO -  _run_next_test@D:\task_174906583785244\build\tests\xpcshell\head.js:1759:38
[task 2025-06-04T20:10:47.933+00:00] 20:10:47     INFO -  run@D:\task_174906583785244\build\tests\xpcshell\head.js:808:9
[task 2025-06-04T20:10:47.934+00:00] 20:10:47     INFO -  _do_main@D:\task_174906583785244\build\tests\xpcshell\head.js:245:6
[task 2025-06-04T20:10:47.934+00:00] 20:10:47     INFO -  _execute_test@D:\task_174906583785244\build\tests\xpcshell\head.js:596:5
[task 2025-06-04T20:10:47.934+00:00] 20:10:47     INFO -  @-e:1:1
[task 2025-06-04T20:10:47.935+00:00] 20:10:47     INFO -  exiting test

the ResetPrefs error seems to be coming from here which implies the preferences system was never stood up correctly.

The "ProfLDS" failure I have no clue about.

Comment 5

[Kershaw Chang [:kershaw]]

the ResetPrefs error seems to be coming from here which implies the preferences system was never stood up correctly.

No, based on the condition below, the issue seems to be that ResetPrefs is not allowed when gSharedMap is non-null:

   if (gSharedMap) {
    return NS_ERROR_NOT_AVAILABLE;
  }

The gSharedMap is created by SharedPreferenceSerializer::SerializeToSharedMemory when the socket process is launched.

I'm not an expert on the preferences code and haven’t looked deeply into it, so my suggestion is to skip these failing tests in the X-spi-nw suite for now.

Comment 6

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1969102 - Skip some libpref tests when socketprocess is in use r?gstoll!,kershaw!

Comment 7

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1969102 - Skip some networking tests when socketprocess is in use r?kershaw!

Comment 8

[Beth Rennie [:beth] (she/her)]

A new try with those failing tasks skipped: https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=143136

Comment 9

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1969102 - Skip some telemetry tests when socketprocess is in use r?chutten!,kershaw!

Comment 10

[Beth Rennie [:beth] (she/her)]

Some more disabled tests: https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=143157