Open Bug 1712329 Opened 7 months ago Updated 16 days ago

run some tests against msix packages

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(Not tracked)

People

(Reporter: bhearsum, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fidedi-tikka] )

Attachments

(13 obsolete files)

So far, we know we want to run xpcshell, browser-chrome, and mochitest-media.

This will involve:

  • Adding msix support to mozinstall for installation
  • Adding some glue code to mozharness to be able to tell it to use the msix packages instead of target.zip
  • Taskgraph work to get the tests scheduled

Tests must run against signed builds.

We also need to ensure that the tests will even run against our Windows 10 machines, which are version 1803.

Other notes I'm dropping for myself:

  • We may need to look into enabling Developer Mode (I don't think this is needed since we're testing signed builds, but it's a small possibility still).
  • Our "Fake CA" is already installed as a root on these machines, so our non-production signing should be seen as valid.
  • If we end up needing to run tests in non-VM pools, we have to watch our for capacity issues.
  • INSTALLER_PATH in the mozharness configs probably needs to be overridden, as it currently assumes a zip is used: https://searchfox.org/mozilla-central/source/testing/mozharness/configs/unittests/win_unittest.py#13

They're going to get out of date horribly quickly, but I'm going to post a patch stack so there's some record of what I'm planning for this that's not on my laptop.

These patches here are the latest I have; they're built on top of the gecko patches from bug 1712328 (which means it requires signingscript support to function as well).

Depends on D119213

Attachment #9223035 - Attachment is obsolete: true
Attachment #9223036 - Attachment is obsolete: true
Attachment #9223037 - Attachment is obsolete: true
Attachment #9223038 - Attachment is obsolete: true
Group: partner-confidential
Depends on: 1712328

This has been deprioritized a bit, so I'm setting it aside for the moment.

I've updated the phab revisions to base off of the signing stack (ending with https://phabricator.services.mozilla.com/D119899).

My latest try push is failing because some more mozharness work is needed: https://treeherder.mozilla.org/jobs?repo=try&revision=a1b071193a3d04f84b2ccdcd5811db309b39b44f&selectedTaskRun=Ls9P4O0-T3SffbOzMfvB6A.0

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

Aryx, do you have any thoughts on how we should schedule these?

Flags: needinfo?(aryx.bugmail)

How about scheduling the tests daily at 1am UTC / 9pm EDT / 6pm PDT? The bottleneck is likely the Windows testing machines, not the Linux builders. This would provide ample time to identify issues before the next Nightly.

Flags: needinfo?(aryx.bugmail)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #13)

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

I may be missing something, but have we thought about adding these tests to the nightly graph and having them depend on the langpacks directly?

Alternately we can cron something that finds the latest nightly decision task in the index, and adds the tests via add-new-jobs and/or the snowman model. It seems easier to run them in the nightly graph.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #15)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #13)

I'm still working on the implementation of this, but one thing that's notable is that running these tests will require us to do all langpack build and signing on Linux -- something we currently only do for nightlies. https://treeherder.mozilla.org/jobs?repo=try&revision=b80bd587cdeb51fa3ed9e5602fdbb8fa0c070547 has an example of all the tasks needed.

Because of this, I'm not sure what sort of frequency we should run them on, or how we should determine whether they should be scheduled. Even if we wanted to only run them on revisions that had already scheduled nightlies, they would end up rebuilding the same tasks unless we added support for finding them in a nightly graph (not trivial, and probably quiite messy).

I may be missing something, but have we thought about adding these tests to the nightly graph and having them depend on the langpacks directly?

Alternately we can cron something that finds the latest nightly decision task in the index, and adds the tests via add-new-jobs and/or the snowman model. It seems easier to run them in the nightly graph.

This is a pretty reasonable option I think, and obviously avoids any unnecessary rebuilding of langpacks. It presumes that we won't run them at all on autoland if I understand correctly (which I guess was an unstated part of my questions here). Aryx, what do you think about this idea?

Flags: needinfo?(aryx.bugmail)

Okay, let's go with the straightforward option to have them in Nightly graph.

Flags: needinfo?(aryx.bugmail)

This is proving to be very difficult. I've gotten to a point where the test jobs can launch Firefox -- but it immediately crashes due to the fact that the environment variables we set aren't inherited when launching the installed MSIX application. (Most notably, MOZ_DISABLE_NONLOCAL_CONNECTIONS is needed to avoid this assert: https://searchfox.org/mozilla-central/source/js/xpconnect/src/nsXPConnect.cpp#1035)

I've worked this with nalexander a bit, and as best we can tell there's a two ways to launch an installed MSIX package:

  1. Through start or explorer, using the shell: protocol. Eg: explorer.exe shell:AppsFolder\Mozilla.MozillaFirefoxNightly_abcdef123456!FIREFOX. This does not appear to allow for arguments to be passed to firefox.exe (which means we can't enable marionette, and do other necessary things), nor does it seem to inherit environment variables (google suggests there may be ways to make that work, but it's not worth pursuing given the issue with arguments.
  2. Using the Invoke-CommandInDesktopPackage cmdlet. This allows us to pass arguments, but as best I can tell, there's absolutely no way to set environment variables.

Assuming I haven't missed something, and all of the above is correct, the most obvious path forward to me would be to route around the need for environment variables -- most likely by adding a way to use arguments or prefs in place of all of the ones we depend on.

It is not possible to launch an application installed through an MSIX package directly -- it must be done through a helper like this.

Depends on D127703

Supports testing of Firefox installed via an MSIX package, where we have no way to set environment variables before launching

Depends on D127704

I managed to work through the startup issues and actually get some tests running. However, I learned that all of our test result processing relies on having access stdout -- which we do not have when launching with Invoke-CommandInDesktopPackage, nor does there seem to be a way to get it after the fact.

It's certainly possible to work around this, eg: by altering both the javascript and python sides of the test harnesses to route through a file instead, but this work has greatly expanded in scope to the point where I'm not sure it's worthwhile pursuing this at the moment.

This is getting backburned for now, for the reasons outlined in the previous comment.

Assignee: bhearsum → nobody
Attachment #9244855 - Attachment is obsolete: true
Attachment #9244613 - Attachment is obsolete: true
Attachment #9244616 - Attachment is obsolete: true
Attachment #9244615 - Attachment is obsolete: true
Attachment #9229942 - Attachment is obsolete: true
Attachment #9229941 - Attachment is obsolete: true
Attachment #9229940 - Attachment is obsolete: true
Attachment #9229939 - Attachment is obsolete: true
Whiteboard: [fidedi-tikka]
Whiteboard: [fidedi-tikka] → [fidedi-tikka]

I happened to be working on some launch-related issues (Bug 1733284), which led me to dig into the launcher process, which led me to investigate interactions with the environment variable behaviour Ben was witnessing. It's complicated, but there seems to be something specific to launching in the package environment that ignores the launching/parent process' environment. Here's one interpretation: per https://github.com/microsoftdocs/msix-docs/blob/master/msix-src/msix-container.md, "Apps that are packaged using MSIX run in a lightweight app container." I wonder if what's happening is that the app container (which is not obviously a separate process, and may not even be involved for these Desktop apps) is started with the "system" environment, much like how on macOS .app bundles inherit the system environment and not the launching shell's environment.

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

(In reply to Nick Alexander :nalexander [he/him] from comment #26)

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

Wow, nice find! If we were to take this approach, does this mean that we need to package our entire test suite and everything it depends on in the MSIX bundle? That seems possible, although it would mean we cannot test the exact MSIX packages we ship.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #27)

(In reply to Nick Alexander :nalexander [he/him] from comment #26)

What I do see is that after Invoke-CommandInDesktopPackage -PreventBreakaway -Command powershell, subsequently launched subshells (and Firefox processes) do witness the powershell modified environment. That suggests that we can launch a shell in the package environment, configure its environment, and then launch Firefox from within that. This doesn't help capture output in a straightforward manner, AFAICT. But perhaps we can launch the whole test apparatus in the package environment? That is, we run everything -- the Python code that configures the environment and captures stdout and the Firefox that reads the environment and outputs to stdout -- "in the app package bubble". It might work...

Wow, nice find! If we were to take this approach, does this mean that we need to package our entire test suite and everything it depends on in the MSIX bundle? That seems possible, although it would mean we cannot test the exact MSIX packages we ship.

No, I don't think so. I don't really understand the "package bubble", but it's clear that it can reach out to the regular filesystem in many situations. (We still might find VFS issues, of course.) The package bubble can invoke many tools: cmd, powershell, regedit, etc (that's how you're supposed to witness the virtualization of the registry, for example). I don't see why the package bubble couldn't invoke ./mach test ... from a directory outside the MSIX.

Attachment #9244614 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.