1937434 - [c-c msOS] perma test failures in toolkit/mozapps/update/tests/ for Mac (toolkit/mozapps/update/tests/unit_aus_update/multiSessionInstallLockout.js, marAppApplyUpdateSuccess.js, marAppApplyUpdateStageSuccess.js)

Status: RESOLVED FIXED

(Thunderbird :: Upstream Synchronization, defect)

Description

[Geoff Lankow (:darktrojan)]

We've got some new failures on Mac:

• toolkit/mozapps/update/tests/unit_aus_update/multiSessionInstallLockout.js
• toolkit/mozapps/update/tests/unit_base_updater/marAppApplyUpdateSuccess.js
• toolkit/mozapps/update/tests/unit_base_updater/marAppApplyUpdateStageSuccess.js

Here's an example failure log.

They'll be from bug 1907120, bug 1907126, or bug 1907127. I tried adding the preferences in this revision which changed the behaviour (the OS gave me a warning about launching an application) but that didn't really help and I gave up.

Comment 1

[Geoff Lankow (:darktrojan)]

Robin, do you have any idea why these failures might be happening when the tests are run in Thunderbird?

Comment 2

[Robin Steuber (she/her) [:bytesized]]

Interestingly, I had problems with these same three tests, I believe only on macOS, when developing these patches. But the thing is that I backtracked on the issue that caused this in Firefox and took a different approach. But it seems like some related problem still exists on Thunderbird for some reason.

These three tests are a bit unusual. multiSessionInstallLockout.js is the only one test that runs runUpdateUsingApp more than once. And the other two are the only tests that use the removeOriginalFile feature and they use it to remove bits of the the updater binary. This has a slightly complicated interaction with the function that copies the updater in to place, copyTestUpdaterToBinDir.

Originally, I ended up with problems where, on macOS, I'd end up with a partial updater app on the second call to runUpdateUsingApp in multiSessionInstallLockout.js. Since the updater app directory existed, this line prevented the app from being rewritten. I never did figure out why the directory already existed in that case; seems like it shouldn't have. But when I tried to remove that line from copyTestUpdaterToBinDir and unconditionally overwrite the updater app, that caused the marAppApplyUpdate(Stage)?Success.js tests to fail because they reran that function after calling removeOriginalFile and suddenly the removed file was back. Because so many other tests call into the functions that those two call into, fixing them to not call that again seemed like a problem. So I reverted copyTestUpdaterToBinDir and added this line instead, which prevented that part of the initialization from being skipped (here) when multiSessionInstallLockout.js repeats.

I'm pretty confused about what's going on in this case. The linked test log shows that we are waiting for the application process launched with runUpdateUsingApp to run this, which it generally runs before it exits for the final time. I can't really figure out what, if any, connection there is between the previous problems and this one. Given that other tests use runUpdateUsingApp besides these 3, it seems unlikely to be a problem with that code specifically. It seems more likely that you are having some kind of a problem with bit of the updater app being missing and consequently the thing not running at all, like the problem I had the first time. But I can't figure out why just by reading the logs.

Comment 3

[Intermittent Failures Robot]

Summary

19 failures in 3774 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 19

Platform and build breakdown:

• macosx1015-64-qr: 14
  • opt: 7
  • debug: 7
• macosx1015-64-shippable-qr: 5
  • opt: 5

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wr-dc1-p	wr-dc2-o	wr-dc3-c	wr-dc3-c-s	xorig	xorig-s
macosx1015-64-qr/opt																					7
macosx1015-64-qr/debug																					7
macosx1015-64-shippable-qr/opt																																5

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2024-12-16&endday=2024-12-22&tree=all

Comment 4

[Intermittent Failures Robot]

Summary

6 failures in 1319 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 6

Platform and build breakdown:

• macosx1015-64-qr: 5
  • opt: 3
  • debug: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

Table

	no_variant
macosx1015-64-qr/opt	3
macosx1015-64-qr/debug	2
macosx1015-64-shippable-qr/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2024-12-30&endday=2025-01-05&tree=all

Comment 5

[Intermittent Failures Robot]

Summary

13 failures in 3682 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 13

Platform and build breakdown:

• macosx1015-64-qr: 9
  • opt: 5
  • debug: 4
• macosx1015-64-shippable-qr: 4
  • opt: 4

Table

	no_variant
macosx1015-64-qr/opt	5
macosx1015-64-qr/debug	4
macosx1015-64-shippable-qr/opt	4

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2025-01-06&endday=2025-01-12&tree=all

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

going to disable this test on macosx1100 for perma timeout in bug 1942377

Comment 7

[Intermittent Failures Robot]

Summary

19 failures in 4508 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 19

Platform and build breakdown:

• macosx1015-64-qr: 19
  • opt: 13
  • debug: 6

Table

	no_variant
macosx1015-64-qr/-/opt	13
macosx1015-64-qr/-/debug	6

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2025-01-13&endday=2025-01-19&tree=all

Comment 8

[Magnus Melin [:mkmelin]]

toolkit/mozapps/update/tests/unit_update_binary/marAppApplyUpdateStageSuccess.js and toolkit/mozapps/update/tests/unit_aus_update/multiSessionInstallLockout.js are still permafailing.

Comment 9

[Intermittent Failures Robot]

Summary

18 failures in 756 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 17
• mozilla-beta: 1

Platform and build breakdown:

• macosx1015-64-qr: 17
  • opt: 11
  • debug: 6
• macosx1100-aarch64-qr: 1
  • opt: 1

Table

	no_variant
macosx1015-64-qr--/opt	11
macosx1015-64-qr--/debug	6
macosx1100-aarch64-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2025-01-23&endday=2025-01-23&tree=all

Comment 10

[Intermittent Failures Robot]

Summary

43 failures in 4035 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #42 most frequent failure this week.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• comm-central: 41
• mozilla-beta: 2

Platform and build breakdown:

• macosx1015-64-qr: 41
  • opt: 27
  • debug: 14
• macosx1100-aarch64-qr: 2
  • opt: 2

Table

	no_variant
macosx1015-64-qr--/opt	27
macosx1015-64-qr--/debug	14
macosx1100-aarch64-qr--/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2025-01-20&endday=2025-01-26&tree=all

Comment 11

[Geoff Lankow (:darktrojan)]

I have figured out what's wrong. The update .app we're trying to run has "firefox" wired in as the name of the executable, but the actual executable is "thunderbird". Checking with a try run then I'll push a patch.

Comment 12

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1937434 - Don't hard-wire the executable name in app update test packages. r=bytesized

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4da4b44a870
Don't hard-wire the executable name in app update test packages. r=bytesized,application-update-reviewers

Comment 14

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a4da4b44a870

Comment 15

[Intermittent Failures Robot]

Summary

11 failures in 4566 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 11

Platform and build breakdown:

• macosx1015-64-qr: 11
  • opt: 7
  • debug: 4

Table

	no_variant
macosx1015-64-qr--/opt	7
macosx1015-64-qr--/debug	4

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1937434&startday=2025-01-27&endday=2025-02-02&tree=all