1724466 - Include MSIX package(s) on archive.mozilla.org listing

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Uploading, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

archive.mozilla.org lists MSI packages in per-locale locations like
https://archive.mozilla.org/pub/firefox/releases/85.0/win64/en-US/Firefox%20Setup%2085.0.msi.

This ticket tracks listing per-architecture MSIX packages in multi-locale locations like
https://archive.mozilla.org/pub/firefox/releases/85.0/win64/Firefox%20x64%2085.0.msix
or
https://archive.mozilla.org/pub/firefox/releases/85.0/win64/multi/Firefox%20x64%2085.0.msix

Eventually, we may produce a multi-architecture bundle and list

https://archive.mozilla.org/pub/firefox/releases/85.0/Firefox%2085.0.msix

Comment 1

[Nick Alexander :nalexander [he/him]]

bhearsum: could you re-categorize this as appropriate? Thanks!

Comment 2

[[e-mail] (:bhearsum)]

Do we have a deadline for this?

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #2)

Do we have a deadline for this?

Not that I'm aware of. I am just capturing tasks that I am aware of in this space.

Comment 4

[Nick Alexander :nalexander [he/him]]

In a different medium, :bhearsum asked, "When you say buildid, that is the same buildid that ends up in application.ini, right?"

Yes. For Nightly builds only (at this time), the build ID (fished from application.ini!) is mangled into the Windows version code (code here, tests here).

:bhearusm also asked: "Can you give me an example of where you'd want it published for both Nightly & Release? I'm expecting something like

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/firefox-92.0a1.en-US.win32.installer.msi

for Nightly, and something like

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/en-US/Firefox%20Setup%2091.0.msi

for Release. I do wonder if we need the latter to go in a multi directory instead of en-US though."

For release, your suggested scheme matches my proposed scheme in #c0 (up to the .msix extension). On disk, these packages are installed into locations like

C:\Program Files\WindowsApps\Mozilla.Firefox.Nightly_93.2021.812.435_x64__5x4grbbqzn2q4\VFS\ProgramFiles\Firefox Nightly Package Root\firefox.exe

for Nightly and

C:\Program Files\WindowsApps\Mozilla.Firefox_90.0.0.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\Firefox Package Root\firefox.exe

for Release. I would like to keep the version number and architecture, to the extent possible, in the package name for simplicity. So ideally it would be

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/Mozilla.Firefox.Nightly_93.2021.812.435_x64.msix

for Nightly and

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/multi/Mozilla.Firefox_90.0.0.0_x64.msix

So the app identifier (Mozilla.Firefox[.Nightly,.Beta,.Developer.Edition]) will vary based on channel, and the version number will vary depending on the milestone (and, for Nightly only, the build ID).

:bhearsum: is that clear? I imagine that we'll have to wire that get_embedded_version into the taskgraph in some way to support the beetmover details.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

In a different medium, :bhearsum asked, "When you say buildid, that is the same buildid that ends up in application.ini, right?"

Yes. For Nightly builds only (at this time), the build ID (fished from application.ini!) is mangled into the Windows version code (code here, tests here).

:bhearusm also asked: "Can you give me an example of where you'd want it published for both Nightly & Release? I'm expecting something like

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/firefox-92.0a1.en-US.win32.installer.msi

for Nightly, and something like

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/en-US/Firefox%20Setup%2091.0.msi

for Release. I do wonder if we need the latter to go in a multi directory instead of en-US though."

For release, your suggested scheme matches my proposed scheme in #c0 (up to the .msix extension). On disk, these packages are installed into locations like

C:\Program Files\WindowsApps\Mozilla.Firefox.Nightly_93.2021.812.435_x64__5x4grbbqzn2q4\VFS\ProgramFiles\Firefox Nightly Package Root\firefox.exe

for Nightly and

C:\Program Files\WindowsApps\Mozilla.Firefox_90.0.0.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\Firefox Package Root\firefox.exe

for Release. I would like to keep the version number and architecture, to the extent possible, in the package name for simplicity. So ideally it would be

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/Mozilla.Firefox.Nightly_93.2021.812.435_x64.msix

for Nightly and

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/multi/Mozilla.Firefox_90.0.0.0_x64.msix

This is clear, although I'm not sure why we need to embed the version & arch into the filename for /releases/. If it makes no difference to you, I'd prefer we follow the current convention of only putting them in the leading directories. Not a hard requirement, just a preference.

So the app identifier (Mozilla.Firefox[.Nightly,.Beta,.Developer.Edition]) will vary based on channel, and the version number will vary depending on the milestone (and, for Nightly only, the build ID).

OK, appid and identifier should be easy to wire into taskgraph.

:bhearsum: is that clear? I imagine that we'll have to wire that get_embedded_version into the taskgraph in some way to support the beetmover details.

Yeah, ideally we calculate it at taskgraph generation time. It's possible to do it at runtime (we do it for partials), but that may mean duplicating the logic into https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript, which I'd like to avoid.

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #5)

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

In a different medium, :bhearsum asked, "When you say buildid, that is the same buildid that ends up in application.ini, right?"

Yes. For Nightly builds only (at this time), the build ID (fished from application.ini!) is mangled into the Windows version code (code here, tests here).

:bhearusm also asked: "Can you give me an example of where you'd want it published for both Nightly & Release? I'm expecting something like

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/firefox-92.0a1.en-US.win32.installer.msi

for Nightly, and something like

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/en-US/Firefox%20Setup%2091.0.msi

for Release. I do wonder if we need the latter to go in a multi directory instead of en-US though."

For release, your suggested scheme matches my proposed scheme in #c0 (up to the .msix extension). On disk, these packages are installed into locations like

C:\Program Files\WindowsApps\Mozilla.Firefox.Nightly_93.2021.812.435_x64__5x4grbbqzn2q4\VFS\ProgramFiles\Firefox Nightly Package Root\firefox.exe

for Nightly and

C:\Program Files\WindowsApps\Mozilla.Firefox_90.0.0.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\Firefox Package Root\firefox.exe

for Release. I would like to keep the version number and architecture, to the extent possible, in the package name for simplicity. So ideally it would be

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/Mozilla.Firefox.Nightly_93.2021.812.435_x64.msix

for Nightly and

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/multi/Mozilla.Firefox_90.0.0.0_x64.msix

This is clear, although I'm not sure why we need to embed the version & arch into the filename for /releases/. If it makes no difference to you, I'd prefer we follow the current convention of only putting them in the leading directories. Not a hard requirement, just a preference.

Nothing here is a requirement. If it's onerous, we drop it. If you want just Mozilla.Firefox{.Beta}.msix, fine by me. (Aside: I just moved to differentiate Release from Beta in a few places, because it's so hard to keep them straight otherwise. But we may roll that back to allow testing Beta-as-Release for MSIX.)

So the app identifier (Mozilla.Firefox[.Nightly,.Beta,.Developer.Edition]) will vary based on channel, and the version number will vary depending on the milestone (and, for Nightly only, the build ID).

OK, appid and identifier should be easy to wire into taskgraph.

:bhearsum: is that clear? I imagine that we'll have to wire that get_embedded_version into the taskgraph in some way to support the beetmover details.

Yeah, ideally we calculate it at taskgraph generation time. It's possible to do it at runtime (we do it for partials), but that may mean duplicating the logic into https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript, which I'd like to avoid.

I would be fine doing it at taskgraph time, and then always providing a full version to the script from the taskgraph parameters, but we don't really do that for the build ID (which is used, at least for Nightly revisions). Can we count on the taskgraph parameters having MOZ_BUILD_DATE or similar, so that we can produce the build ID and/or its derivatives itself? This may need a spin-off of its own.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

(In reply to [e-mail] (:bhearsum) from comment #5)

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

In a different medium, :bhearsum asked, "When you say buildid, that is the same buildid that ends up in application.ini, right?"

Yes. For Nightly builds only (at this time), the build ID (fished from application.ini!) is mangled into the Windows version code (code here, tests here).

:bhearusm also asked: "Can you give me an example of where you'd want it published for both Nightly & Release? I'm expecting something like

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/firefox-92.0a1.en-US.win32.installer.msi

for Nightly, and something like

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/en-US/Firefox%20Setup%2091.0.msi

for Release. I do wonder if we need the latter to go in a multi directory instead of en-US though."

For release, your suggested scheme matches my proposed scheme in #c0 (up to the .msix extension). On disk, these packages are installed into locations like

C:\Program Files\WindowsApps\Mozilla.Firefox.Nightly_93.2021.812.435_x64__5x4grbbqzn2q4\VFS\ProgramFiles\Firefox Nightly Package Root\firefox.exe

for Nightly and

C:\Program Files\WindowsApps\Mozilla.Firefox_90.0.0.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\Firefox Package Root\firefox.exe

for Release. I would like to keep the version number and architecture, to the extent possible, in the package name for simplicity. So ideally it would be

https://archive.mozilla.org/pub/firefox/nightly/2021/08/2021-08-09-09-33-20-mozilla-central/Mozilla.Firefox.Nightly_93.2021.812.435_x64.msix

for Nightly and

https://archive.mozilla.org/pub/firefox/releases/91.0/win64/multi/Mozilla.Firefox_90.0.0.0_x64.msix

This is clear, although I'm not sure why we need to embed the version & arch into the filename for /releases/. If it makes no difference to you, I'd prefer we follow the current convention of only putting them in the leading directories. Not a hard requirement, just a preference.

Nothing here is a requirement. If it's onerous, we drop it. If you want just Mozilla.Firefox{.Beta}.msix, fine by me. (Aside: I just moved to differentiate Release from Beta in a few places, because it's so hard to keep them straight otherwise. But we may roll that back to allow testing Beta-as-Release for MSIX.)

OK, I'm going to go with these ones, in that case.

So the app identifier (Mozilla.Firefox[.Nightly,.Beta,.Developer.Edition]) will vary based on channel, and the version number will vary depending on the milestone (and, for Nightly only, the build ID).

OK, appid and identifier should be easy to wire into taskgraph.

:bhearsum: is that clear? I imagine that we'll have to wire that get_embedded_version into the taskgraph in some way to support the beetmover details.

Yeah, ideally we calculate it at taskgraph generation time. It's possible to do it at runtime (we do it for partials), but that may mean duplicating the logic into https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript, which I'd like to avoid.

I would be fine doing it at taskgraph time, and then always providing a full version to the script from the taskgraph parameters, but we don't really do that for the build ID (which is used, at least for Nightly revisions). Can we count on the taskgraph parameters having MOZ_BUILD_DATE or similar, so that we can produce the build ID and/or its derivatives itself? This may need a spin-off of its own.

Huh, I'm a bit surprised we don't do it for build ID. I see moz_build_date in the taskgraph decision task parameters, eg on https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=decision&selectedTaskRun=BEupjOgyQH239tkCRvrTQw.0, and some usage of it in taskgraph already: https://searchfox.org/mozilla-central/search?q=moz_build_date&case=true&path=taskcluster

Comment 8

[Justin Wood (:Callek)]

(In reply to [e-mail] (:bhearsum) from comment #7)

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

(In reply to [e-mail] (:bhearsum) from comment #5)

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

:bhearsum: is that clear? I imagine that we'll have to wire that get_embedded_version into the taskgraph in some way to support the beetmover details.

Yeah, ideally we calculate it at taskgraph generation time. It's possible to do it at runtime (we do it for partials), but that may mean duplicating the logic into https://github.com/mozilla-releng/scriptworker-scripts/tree/master/beetmoverscript, which I'd like to avoid.

I would be fine doing it at taskgraph time, and then always providing a full version to the script from the taskgraph parameters, but we don't really do that for the build ID (which is used, at least for Nightly revisions). Can we count on the taskgraph parameters having MOZ_BUILD_DATE or similar, so that we can produce the build ID and/or its derivatives itself? This may need a spin-off of its own.

Huh, I'm a bit surprised we don't do it for build ID. I see moz_build_date in the taskgraph decision task parameters, eg on https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=decision&selectedTaskRun=BEupjOgyQH239tkCRvrTQw.0, and some usage of it in taskgraph already: https://searchfox.org/mozilla-central/search?q=moz_build_date&case=true&path=taskcluster

It is generated as part of the decision task. And this is then passed down from the taskgraph into the build tasks. https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/decision.py#345-350 Its generally based on the push information, but there are fallbacks of course

Comment 9

[[e-mail] (:bhearsum)]

Attachment: Bug 1724466: RELENG-630: Add support for publishing MSIX packages to archive.mozilla.org. r=#releng-reviewers!

This is a lot of copy/paste, but there's a couple of notable things:

• MSIX packages are multilocale. I chose to create a new multi folder for them rather than put them in the en-US folder.
• Due to the above, the packages we ship come out of a new repackage job (repackage-signing-shippable-l10n-msix)

Comment 10

[[e-mail] (:bhearsum)]

Attachment: Bug 1724466: RELENG-630: manage msix bouncer entries for nightly & releases. r=#releng-reviewers!

99% copy/paste. The only notable things are:

• There's no l10n entry because msix packages are multilocale
• Similarly, add_locales needed to be flipped to False for msix packages

Depends on D125276

Comment 11

[[e-mail] (:bhearsum)]

Attachment: Bug 1724466: RELENG-635: test msix bouncer products as part of bouncer-check. r=#releng-reviewers!

Depends on D126949

Comment 12

[[e-mail] (:bhearsum)]

The patches I just posted are using the MSI-style filenames for now (see http://ftp.stage.mozaws.net/pub/firefox/releases/94.0b2/win32/multi/ for an example). We can follow-up to change that, but I suggest we go with it at the moment to make sure we're publishing something.

I also wonder if enough has changed since the earlier discussion that including the 4 part version doesn't make sense anymore.

Nick, let me know what you think.

Comment 13

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #12)

The patches I just posted are using the MSI-style filenames for now (see http://ftp.stage.mozaws.net/pub/firefox/releases/94.0b2/win32/multi/ for an example). We can follow-up to change that, but I suggest we go with it at the moment to make sure we're publishing something.

I would really like to drop the Setup part, because this isn't an installer, it's a package.

I also wonder if enough has changed since the earlier discussion that including the 4 part version doesn't make sense anymore.

I don't have a strong opinion here. I think the domain (i.e., the MSIX industrial complex) would much prefer Mozilla.MozillaFirefox.X.Y.Z.0_arch.msix but it doesn't fit with the rest of the things we do so I'm not going to push too hard for it.

I think publishing something is a great first step that we can iterate on. In fact, I think we're going to have a period where our package identifiers aren't stable as we sort out some ecosystem issues, so early adopters will have churn on many fronts.

Great work!

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f37d3eb31e8a
RELENG-630: Add support for publishing MSIX packages to archive.mozilla.org. r=releng-reviewers,jmaher
https://hg.mozilla.org/integration/autoland/rev/c76cd032500c
RELENG-630: manage msix bouncer entries for nightly & releases. r=releng-reviewers,jmaher
https://hg.mozilla.org/integration/autoland/rev/be46c9023268
RELENG-635: test msix bouncer products as part of bouncer-check. r=releng-reviewers,jmaher

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/f37d3eb31e8a
https://hg.mozilla.org/mozilla-central/rev/c76cd032500c
https://hg.mozilla.org/mozilla-central/rev/be46c9023268