1712329 - (msix-testing) run some tests against msix packages

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[[e-mail] (:bhearsum)]

So far, we know we want to run xpcshell, browser-chrome, and mochitest-media.

This will involve:

• Adding msix support to mozinstall for installation
• Adding some glue code to mozharness to be able to tell it to use the msix packages instead of target.zip
• Taskgraph work to get the tests scheduled

Tests must run against signed builds.

We also need to ensure that the tests will even run against our Windows 10 machines, which are version 1803.

Comment 1

[[e-mail] (:bhearsum)]

Other notes I'm dropping for myself:

• We may need to look into enabling Developer Mode (I don't think this is needed since we're testing signed builds, but it's a small possibility still).
• Our "Fake CA" is already installed as a root on these machines, so our non-production signing should be seen as valid.
• If we end up needing to run tests in non-VM pools, we have to watch our for capacity issues.
• INSTALLER_PATH in the mozharness configs probably needs to be overridden, as it currently assumes a zip is used: https://searchfox.org/mozilla-central/source/testing/mozharness/configs/unittests/win_unittest.py#13

Comment 2

[[e-mail] (:bhearsum)]

They're going to get out of date horribly quickly, but I'm going to post a patch stack so there's some record of what I'm planning for this that's not on my laptop.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: 0001-Add-support-for-MSIX-packages-to-mozinstall.patch

Comment 4

[[e-mail] (:bhearsum)]

Attachment: 0002-Allow-test-kind-s-mozharness.requires-signed-builds-.patch

Comment 5

[[e-mail] (:bhearsum)]

Attachment: 0003-Enable-xpcshell-msix-variant-testing.patch

Comment 6

[[e-mail] (:bhearsum)]

Attachment: 0004-Enable-mochitest-browser-chrome-and-media-tests-for-.patch

Comment 7

[[e-mail] (:bhearsum)]

These patches here are the latest I have; they're built on top of the gecko patches from bug 1712328 (which means it requires signingscript support to function as well).

Comment 8

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Add support for MSIX packages to mozinstall

Comment 9

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Allow test kind's mozharness.requires-signed-builds and target to be keyed by variant

Depends on D119212

Comment 10

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Enable xpcshell msix variant testing

Depends on D119213

Comment 11

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Enable mochitest browser-chrome and media tests for msix variant

Depends on D119214

Comment 12

[[e-mail] (:bhearsum)]

This has been deprioritized a bit, so I'm setting it aside for the moment.

I've updated the phab revisions to base off of the signing stack (ending with https://phabricator.services.mozilla.com/D119899).

My latest try push is failing because some more mozharness work is needed: https://treeherder.mozilla.org/jobs?repo=try&revision=a1b071193a3d04f84b2ccdcd5811db309b39b44f&selectedTaskRun=Ls9P4O0-T3SffbOzMfvB6A.0

Comment 13

[[e-mail] (:bhearsum)]

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

Aryx, do you have any thoughts on how we should schedule these?

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

How about scheduling the tests daily at 1am UTC / 9pm EDT / 6pm PDT? The bottleneck is likely the Windows testing machines, not the Linux builders. This would provide ample time to identify issues before the next Nightly.

Comment 15

[Aki Sasaki (not active)]

(In reply to [e-mail] (:bhearsum) from comment #13)

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

I may be missing something, but have we thought about adding these tests to the nightly graph and having them depend on the langpacks directly?

Alternately we can cron something that finds the latest nightly decision task in the index, and adds the tests via add-new-jobs and/or the snowman model. It seems easier to run them in the nightly graph.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #15)

(In reply to [e-mail] (:bhearsum) from comment #13)

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

I may be missing something, but have we thought about adding these tests to the nightly graph and having them depend on the langpacks directly?

Alternately we can cron something that finds the latest nightly decision task in the index, and adds the tests via add-new-jobs and/or the snowman model. It seems easier to run them in the nightly graph.

This is a pretty reasonable option I think, and obviously avoids any unnecessary rebuilding of langpacks. It presumes that we won't run them at all on autoland if I understand correctly (which I guess was an unstated part of my questions here). Aryx, what do you think about this idea?

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Okay, let's go with the straightforward option to have them in Nightly graph.

Comment 18

[[e-mail] (:bhearsum)]

This is proving to be very difficult. I've gotten to a point where the test jobs can launch Firefox -- but it immediately crashes due to the fact that the environment variables we set aren't inherited when launching the installed MSIX application. (Most notably, MOZ_DISABLE_NONLOCAL_CONNECTIONS is needed to avoid this assert: https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp#1035)

I've worked this with nalexander a bit, and as best we can tell there's a two ways to launch an installed MSIX package:

1. Through start or explorer, using the shell: protocol. Eg: explorer.exe shell:AppsFolder\Mozilla.MozillaFirefoxNightly_abcdef123456!FIREFOX. This does not appear to allow for arguments to be passed to firefox.exe (which means we can't enable marionette, and do other necessary things), nor does it seem to inherit environment variables (google suggests there may be ways to make that work, but it's not worth pursuing given the issue with arguments.
2. Using the Invoke-CommandInDesktopPackage cmdlet. This allows us to pass arguments, but as best I can tell, there's absolutely no way to set environment variables.

Assuming I haven't missed something, and all of the above is correct, the most obvious path forward to me would be to route around the need for environment variables -- most likely by adding a way to use arguments or prefs in place of all of the ones we depend on.

Comment 19

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Use correct build signing tasks for msix test dependencies

Depends on D119213

Comment 20

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Ensure msix tests show up on try without --full

Depends on D127701

Comment 21

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Support XPIs existing in subdirectories in mozprofile.addons

This case starting existing with https://phabricator.services.mozilla.com/D126174

Depends on D119215

Comment 22

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Launch MSIX-installed Firefox through Invoke-CommandInDesktopPackage cmdlet

It is not possible to launch an application installed through an MSIX package directly -- it must be done through a helper like this.

Depends on D127703

Comment 23

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Allow disabling nonlocal connections via a pref, in addition to the existing environment variable

Supports testing of Firefox installed via an MSIX package, where we have no way to set environment variables before launching

Depends on D127704

Comment 24

[[e-mail] (:bhearsum)]

I managed to work through the startup issues and actually get some tests running. However, I learned that all of our test result processing relies on having access stdout -- which we do not have when launching with Invoke-CommandInDesktopPackage, nor does there seem to be a way to get it after the fact.

It's certainly possible to work around this, eg: by altering both the javascript and python sides of the test harnesses to route through a file instead, but this work has greatly expanded in scope to the point where I'm not sure it's worthwhile pursuing this at the moment.

Comment 25

[[e-mail] (:bhearsum)]

This is getting backburned for now, for the reasons outlined in the previous comment.

Comment 26

[Nick Alexander :nalexander [he/him]]

I happened to be working on some launch-related issues (Bug 1733284), which led me to dig into the launcher process, which led me to investigate interactions with the environment variable behaviour Ben was witnessing. It's complicated, but there seems to be something specific to launching in the package environment that ignores the launching/parent process' environment. Here's one interpretation: per https://github.com/microsoftdocs/msix-docs/blob/master/msix-src/msix-container.md, "Apps that are packaged using MSIX run in a lightweight app container." I wonder if what's happening is that the app container (which is not obviously a separate process, and may not even be involved for these Desktop apps) is started with the "system" environment, much like how on macOS .app bundles inherit the system environment and not the launching shell's environment.

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

Comment 27

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #26)

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

Wow, nice find! If we were to take this approach, does this mean that we need to package our entire test suite and everything it depends on in the MSIX bundle? That seems possible, although it would mean we cannot test the exact MSIX packages we ship.

Comment 28

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #27)

(In reply to Nick Alexander :nalexander [he/him] from comment #26)

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

Wow, nice find! If we were to take this approach, does this mean that we need to package our entire test suite and everything it depends on in the MSIX bundle? That seems possible, although it would mean we cannot test the exact MSIX packages we ship.

No, I don't think so. I don't really understand the "package bubble", but it's clear that it can reach out to the regular filesystem in many situations. (We still might find VFS issues, of course.) The package bubble can invoke many tools: cmd, powershell, regedit, etc (that's how you're supposed to witness the virtualization of the registry, for example). I don't see why the package bubble couldn't invoke ./mach test ... from a directory outside the MSIX.

Comment 29

[[e-mail] (:bhearsum)]

Taking another stab at this.

Comment 30

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: linting pass on files I will be touching r?ahal!

Comment 31

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Add support for MSIX packages to mozinstall r?jmaher!

Depends on D164075

Comment 32

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: linting pass on files I will be touching r?ahal!

Comment 33

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Add support for MSIX packages to mozinstall r?jmaher!

Depends on D164077

Comment 34

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Support XPIs existing in subdirectories in mozprofile.addons r?jmaher!

Depends on D164078

Comment 35

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Copy DLLs to test directory for MSIX tests r?jmaher!

We can't copy certutil.exe into the appdir for MSIX, so we have to do pull its dependencies out instead.

Depends on D164079

Comment 36

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Allow test kind's mozharness.requires-signed-builds and target to be keyed by variant r?ahal!

The original version of this patch (https://phabricator.services.mozilla.com/D119213) was much simpler, but even after the big refactor this is mostly just moving a few things around and adding some keyed by support.

Depends on D164080

Comment 37

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Use correct build signing tasks for msix test dependencies r?ahal!

Depends on D164081

Comment 38

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Ensure msix tests show up on try without --full r?ahal!

Depends on D164082

Comment 39

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Enable mochitest-media tests for msix variant r?jmaher!

This test suite seems to be green on try already.

Depends on D164083

Comment 40

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Enable xpcshell msix variant testing

Depends on D164084

Comment 41

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: run xpcshell out of tests directory instead of app directory

Depends on D164085

Comment 42

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Enable mochitest-browser-chrome tests for msix variant r?jmaher!

Depends on D164086

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/91271c9bed91
linting pass on files I will be touching r=ahal

Comment 44

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/91271c9bed91

Comment 45

[Kagami Rosylight [:saschanaz] (they/them)]

Looks like there are still things to be landed.

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8e40be18e99
Add support for MSIX packages to mozinstall r=jmaher
https://hg.mozilla.org/integration/autoland/rev/81a9fa3a88c2
Support XPIs existing in subdirectories in mozprofile.addons r=jmaher
https://hg.mozilla.org/integration/autoland/rev/a937a2551259
Copy DLLs to test directory for MSIX tests r=jmaher
https://hg.mozilla.org/integration/autoland/rev/16b42b3c4f49
Allow test kind's mozharness.requires-signed-builds and target to be keyed by variant r=ahal
https://hg.mozilla.org/integration/autoland/rev/86bc88c293fe
Use correct build signing tasks for msix test dependencies r=ahal
https://hg.mozilla.org/integration/autoland/rev/054c8b42c54d
Ensure msix tests show up on try without --full r=ahal
https://hg.mozilla.org/integration/autoland/rev/6663585b720e
Enable mochitest-media tests for msix variant r=jmaher,ahal

Comment 47

[Marian-Vasile Laza]

Backed out 7 changesets (Bug 1712329) for breaking the decision task landing.
Backout link
Push with failures
Failure Log

Comment 48

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1712329: Enable xpcshell msix variant testing

Depends on D164084

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d249b4468265
Add support for MSIX packages to mozinstall r=jmaher
https://hg.mozilla.org/integration/autoland/rev/71562a7b9060
Support XPIs existing in subdirectories in mozprofile.addons r=jmaher
https://hg.mozilla.org/integration/autoland/rev/e7f1b3c8f532
Copy DLLs to test directory for MSIX tests r=jmaher
https://hg.mozilla.org/integration/autoland/rev/7cce6af9c6b5
Allow test kind's mozharness.requires-signed-builds and target to be keyed by variant r=ahal
https://hg.mozilla.org/integration/autoland/rev/a64c580e31d0
Use correct build signing tasks for msix test dependencies r=ahal
https://hg.mozilla.org/integration/autoland/rev/108f339bf13e
Ensure msix tests show up on try without --full r=ahal
https://hg.mozilla.org/integration/autoland/rev/9812a580faff
Enable mochitest-media tests for msix variant r=jmaher,ahal

Comment 50

[Marian-Vasile Laza]

Backed out 7 changesets (Bug 1712329) for causing mochitest certificate integration failures.
Backout link
Push with failures
Failure Log
Also X2 failure Log

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8020a614b9f3
Add support for MSIX packages to mozinstall r=jmaher
https://hg.mozilla.org/integration/autoland/rev/940cb854ddbc
Support XPIs existing in subdirectories in mozprofile.addons r=jmaher
https://hg.mozilla.org/integration/autoland/rev/fc86a28edb4a
Copy DLLs to test directory for MSIX tests r=jmaher
https://hg.mozilla.org/integration/autoland/rev/7ddeb04c2170
Allow test kind's mozharness.requires-signed-builds and target to be keyed by variant r=ahal
https://hg.mozilla.org/integration/autoland/rev/1be80c5f4dcd
Use correct build signing tasks for msix test dependencies r=ahal
https://hg.mozilla.org/integration/autoland/rev/94176a637685
Ensure msix tests show up on try without --full r=ahal
https://hg.mozilla.org/integration/autoland/rev/90c03d67a2c2
Enable mochitest-media tests for msix variant r=jmaher,ahal

Comment 52

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/8020a614b9f3
https://hg.mozilla.org/mozilla-central/rev/940cb854ddbc
https://hg.mozilla.org/mozilla-central/rev/fc86a28edb4a
https://hg.mozilla.org/mozilla-central/rev/7ddeb04c2170
https://hg.mozilla.org/mozilla-central/rev/1be80c5f4dcd
https://hg.mozilla.org/mozilla-central/rev/94176a637685
https://hg.mozilla.org/mozilla-central/rev/90c03d67a2c2

Comment 53

[[e-mail] (:bhearsum)]

Comment on attachment 9307079 [details]
Bug 1712329: Add support for MSIX packages to mozinstall r?jmaher!

Beta/Release Uplift Approval Request

• User impact if declined: This work adds our first automated tests of MSIX builds. Without it, there's a much higher chance that we ship a major crash in these builds.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): These changes mostly touch test suite code in a way that doesn't affect existing tests.
• String changes made/needed:
• Is Android affected?: No

Comment 54

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: Add support for not running tests by msix variant r?jmaher!

Comment 55

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9307078 [details]
Bug 1712329: linting pass on files I will be touching r?ahal!

Approved for Beta, but it's a bit messy IMO that we're still landing more patches in this bug instead of using this as a metabug for adding more test suites.

Comment 56

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/06d7910b9700
https://hg.mozilla.org/releases/mozilla-beta/rev/edea6c80bd42
https://hg.mozilla.org/releases/mozilla-beta/rev/67d40b48922b
https://hg.mozilla.org/releases/mozilla-beta/rev/a94d5767d9d4
https://hg.mozilla.org/releases/mozilla-beta/rev/4594b21f4cd0
https://hg.mozilla.org/releases/mozilla-beta/rev/2378c8da1387
https://hg.mozilla.org/releases/mozilla-beta/rev/782bf279265b

Comment 57

[[e-mail] (:bhearsum)]

Attachment: Bug 1712329: improve manifestparser error messages r?jmaher!

Comment 58

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/321f8c1cae6f
improve manifestparser error messages r=jmaher

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1e3e02cc3b1f
Add support for not running tests by msix variant r=jmaher
https://hg.mozilla.org/integration/autoland/rev/9e7813271e6b
Enable mochitest-browser-chrome tests for msix variant r=jmaher

Comment 60

[Iulian Moraru]

Backed out for breaking gecko decision task.

Push with failures

Failure log

Backout link

<...>
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 548, in __call__
[task 2022-12-15T00:49:54.351Z]     tests = self._load_manifestparser_manifest(mpath)
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 519, in _load_manifestparser_manifest
[task 2022-12-15T00:49:54.351Z]     mp = TestManifest(
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 824, in __init__
[task 2022-12-15T00:49:54.351Z]     ManifestParser.__init__(self, *args, **kwargs)
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 97, in __init__
[task 2022-12-15T00:49:54.351Z]     self.read(*manifests)
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 292, in read
[task 2022-12-15T00:49:54.351Z]     self._read(here, filename, defaults)
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/manifestparser/manifestparser/manifestparser.py", line 181, in _read
[task 2022-12-15T00:49:54.351Z]     sections, defaults = read_ini(
[task 2022-12-15T00:49:54.351Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/manifestparser/manifestparser/ini.py", line 140, in read_ini
[task 2022-12-15T00:49:54.351Z]     assert (
[task 2022-12-15T00:49:54.351Z] AssertionError: Found duplicate key skip-if in section DEFAULT
[task 2022-12-15T00:49:54.351Z] Traceback (most recent call last):
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 244, in taskgraph_decision
[task 2022-12-15T00:49:54.352Z]     ret = taskgraph_commands["decision"].func(options)
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/main.py", line 633, in decision
[task 2022-12-15T00:49:54.352Z]     taskgraph_decision(options)
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/decision.py", line 235, in taskgraph_decision
[task 2022-12-15T00:49:54.352Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 169, in full_task_graph
[task 2022-12-15T00:49:54.352Z]     return self._run_until("full_task_graph")
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 423, in _run_until
[task 2022-12-15T00:49:54.352Z]     k, v = next(self._run)
[task 2022-12-15T00:49:54.352Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 309, in _run
[task 2022-12-15T00:49:54.352Z]     new_tasks = kind.load_tasks(
<...>

Comment 61

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

improve manifestparser error messages r=jmaher
https://hg.mozilla.org/integration/autoland/rev/321f8c1cae6f650bcbfc39f9375cd7f4cc1f1847
https://hg.mozilla.org/mozilla-central/rev/321f8c1cae6f

Comment 62

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a820147db1c
Add support for not running tests by msix variant r=jmaher
https://hg.mozilla.org/integration/autoland/rev/a7d5ce257fb5
Enable mochitest-browser-chrome tests for msix variant r=jmaher

Comment 63

[Phabricator Automation]

Comment on attachment 9307087 [details]
WIP: Bug 1712329: run xpcshell out of tests directory instead of app directory

Revision D164086 was moved to bug 1805919. Setting attachment 9307087 [details] to obsolete.

Comment 64

[Phabricator Automation]

Comment on attachment 9307333 [details]
WIP: Bug 1712329: Enable xpcshell msix variant testing

Revision D164230 was moved to bug 1805920. Setting attachment 9307333 [details] to obsolete.

Comment 65

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/6a820147db1c
https://hg.mozilla.org/mozilla-central/rev/a7d5ce257fb5

Comment 66

[[e-mail] (:bhearsum)]

mochitest-media and browser-chrome tests have landed and stuck. I'm going to hold off doing any more direct work in this bug, and deal with additional work is specific bugs. Most notably, I'm going to try to resolve the intermittent bug we're hitting, try to get the tests running on autoland, and try to get xpcshell tests running.

Comment 67

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahal, maybe it's time to close this bug?
For more information, please visit BugBot documentation.

Comment 68

[[e-mail] (:bhearsum)]

I agree that this bug has outlived its usefulness. There's one open intermittent that ought to be followed up on at some point - but we are definitely now running some tests against MSIX packages.