1729546 - test failure in toolkit/modules/subprocess/test/xpcshell/test_subprocess.js on apple silicon

Status: RESOLVED FIXED

(Toolkit :: Async Tooling, task, P3)

Description

[[e-mail] (:bhearsum)]

Last week we turned on tests for Apple Silicon (OSX 11.2.3 on new Apple based hardware). We are using the simplified new test config process:
https://firefox-source-docs.mozilla.org/testing/ci-configs/index.html

As the tests are live, we are now filing bugs to help close the loop and hope to fix any issues over the next 7 weeks. As the process outlines, there are tier-3 jobs running on m-c which run these skipped tests and will expect a failure/timeout/crash- if it doesn't fail, then the job will turn orange.

Here is a log showing the failure: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=macosx1100&revision=eac40293649628d32911c909219d514bebb8ce63&test_paths=toolkit%2Fmodules%2Fsubprocess%2Ftest%2Fxpcshell&selectedTaskRun=dmCjMsSzT7aLxeToJB2KsA.0

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Changing severity to N/A because this does not affect users directly.

Comment 2

[:Gijs (he/him)]

This is showing up in triage emails because defects can't be severity: N/A, so making this into a task for now. However...

(In reply to [e-mail] (:bhearsum) from comment #0)

Last week we turned on tests for Apple Silicon (OSX 11.2.3 on new Apple based hardware). We are using the simplified new test config process:
https://firefox-source-docs.mozilla.org/testing/ci-configs/index.html

As the tests are live, we are now filing bugs to help close the loop and hope to fix any issues over the next 7 weeks.

7 weeks have passed - is this fixed now, or was there an investigation that showed what was wrong here?

Here is a log showing the failure: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=macosx1100&revision=eac40293649628d32911c909219d514bebb8ce63&test_paths=toolkit%2Fmodules%2Fsubprocess%2Ftest%2Fxpcshell&selectedTaskRun=dmCjMsSzT7aLxeToJB2KsA.0

This doesn't seem to include errors from test_subprocess.js, perhaps the link is wrong?

(In reply to Mike Conley (:mconley) (:⚙️) from comment #1)

Changing severity to N/A because this does not affect users directly.

Are we sure about this? Is it possible that the test points out something that would break e.g. interaction of webextensions with external apps using subprocess, on M1 macs?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

we don't have any references to this failure anymore, if you want to get a fresh log, you can edit the manifest and push to try:
https://searchfox.org/mozilla-central/source/toolkit/modules/subprocess/test/xpcshell/xpcshell.ini#13

remove or comment out line 13 which references apple_silicon.

and you can push to try:
./mach try fuzzy -q 'test-macosx1100 xpcshell

Comment 4

[Rob Wu [:robwu]]

I suppose that the test failures happen because there is one test task that takes may take an unusual amount of time on resource-constrained hardware: https://searchfox.org/firefox-main/rev/1c6a8b56e6f5f5393ea5ea745ab63ecb67613c93/toolkit/modules/subprocess/test/xpcshell/test_subprocess.js#179-225

A similar issue happened before with ccov, after which the test was skipped on ccov: bug 1509968.

There is another test that copied the same pattern having issues too: bug 1951522
(see also my recent description of fundamental issues with the test: https://bugzilla.mozilla.org/show_bug.cgi?id=1951522#c31 )

I'm going to split this problematic test from the file and remove the skips. The new file may get a bug filed about intermittent failures, but then at least we will know what the extent of the issue is.

Comment 5

[Rob Wu [:robwu]]

Attachment: Bug 1729546 - Move test_subprocess_round_trip_perf to its own file and unskip some

... and add references to another file that copied the test structure,
plus pointers to bug comments highlighting the fragility of these tests
on busy systems.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/afe117bf0e81
https://hg.mozilla.org/integration/autoland/rev/403f8ae7977f
Move test_subprocess_round_trip_perf to its own file and unskip some r=florian

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/403f8ae7977f