1890607 - Publish ARM64/aarch64 Windows MSIX packages to archive.mozilla.org

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

Following up on Bug 1731140, I see Nightly packages being built, for example around https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=79551503d77c8353439905d86ec3033546f0c874&searchStr=repackage&selectedTaskRun=WiJuBxYBR1KCIfUT9Q1aag.0 but it's not clear that the MSIX packages are getting pushed to archive.mozilla.org (let alone the store). In particular, I don't see anything ARM64-y in https://archive.mozilla.org/pub/firefox/nightly/2024/04/2024-04-09-09-39-00-mozilla-central/.

Per :bhearsum's comment:

• We get signing for free because https://searchfox.org/mozilla-central/source/taskcluster/ci/repackage-signing-shippable-l10n-msix/kind.yml is downstream of repackage-shippable-l10n-msix, and has no platform conditions.
• Nightly publishing to archive is handled by https://searchfox.org/mozilla-central/source/taskcluster/ci/beetmover-repackage/kind.yml, which already has the right kind-dependencies and platforms specified
• Store publishing is handled by https://searchfox.org/mozilla-central/source/taskcluster/ci/release-msix-push/kind.yml, which similarly has the right kind-dependencies, and no platform conditions.

Digging deeper, I see a functional ARM64 beetmover task at https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=79551503d77c8353439905d86ec3033546f0c874&searchStr=beetmover-repackage&selectedTaskRun=P7WzzmmUQ1uqMXPtPqtCng.0, but there's no mention of target.installer.msix in there. So something somewhere needs to tell this task to handle MSIX packages.

:bhearsum, can you guide?

Comment 1

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #0)

Following up on Bug 1731140, I see Nightly packages being built, for example around https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=79551503d77c8353439905d86ec3033546f0c874&searchStr=repackage&selectedTaskRun=WiJuBxYBR1KCIfUT9Q1aag.0 but it's not clear that the MSIX packages are getting pushed to archive.mozilla.org (let alone the store). In particular, I don't see anything ARM64-y in https://archive.mozilla.org/pub/firefox/nightly/2024/04/2024-04-09-09-39-00-mozilla-central/.

Per :bhearsum's comment:

• We get signing for free because https://searchfox.org/mozilla-central/source/taskcluster/ci/repackage-signing-shippable-l10n-msix/kind.yml is downstream of repackage-shippable-l10n-msix, and has no platform conditions.
• Nightly publishing to archive is handled by https://searchfox.org/mozilla-central/source/taskcluster/ci/beetmover-repackage/kind.yml, which already has the right kind-dependencies and platforms specified
• Store publishing is handled by https://searchfox.org/mozilla-central/source/taskcluster/ci/release-msix-push/kind.yml, which similarly has the right kind-dependencies, and no platform conditions.

Digging deeper, I see a functional ARM64 beetmover task at https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=79551503d77c8353439905d86ec3033546f0c874&searchStr=beetmover-repackage&selectedTaskRun=P7WzzmmUQ1uqMXPtPqtCng.0, but there's no mention of target.installer.msix in there. So something somewhere needs to tell this task to handle MSIX packages.

:bhearsum, can you guide?

Ah, my bad. I forgot about the nightly and release manifests that taskgraph consumes. Updating those should do the trick.

It looks like other changes will be needed if we want to make them available through Bouncer as well (which I presume we do). See https://phabricator.services.mozilla.com/D126949 and https://phabricator.services.mozilla.com/D126950 for what we do for x86/x64.

Comment 2

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1890607 - Publish ARM64/aarch64 Windows MSIX packages to beetmover and bouncer. r?bhearsum

Comment 3

[Nick Alexander :nalexander [he/him]]

I'm scoping this down to just archive.mozilla.org and the download bouncer; let's handle publishing to the Microsoft Store separately.

Comment 4

[Nick Alexander :nalexander [he/him]]

bhearsum: I recall the last word being that we'd get releng to push this across the line before revisiting publishing to the Microsoft Store. Can we get an assignee that is not me?

Comment 5

[[e-mail] (:bhearsum)]

Apologies for the delay - Johan is helping getting this assigned.

Comment 6

[Johan Lorenzo [:jlorenzo]]

Discussed over Slack. :gbrown will provide support.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1890607 - Publish Windows aarch64 msix to archive.mozilla.org

Comment 8

[Geoff Brown [:gbrown]]

See discussion in D207087: We need to archive to a.m.o first, then revisit bouncer.

Comment 9

[Geoff Brown [:gbrown]]

I ran a staging release against a patch that only adds win64-aarch64 to beetmover (no bouncer changes) and release-bouncer-sub-firefox failed
https://treeherder.mozilla.org/jobs?repo=try&revision=cc2641cc74b575157c52b45cd603abc08c2a1cf3
https://treeherder.mozilla.org/logviewer?job_id=455765954&repo=try&lineNumber=432

2024-04-25 03:45:05,179 - bouncerscript.utils - INFO - Calling location_show?product=Firefox-127.0b1-msix-SSL with data: {}
2024-04-25 03:45:05,179 - bouncerscript.utils - INFO - Performing a GET request to https://dev.bounceradmin.nonprod.webservices.mozgcp.net/api/location_show?product=Firefox-127.0b1-msix-SSL with kwargs {'timeout': 60}
2024-04-25 03:45:05,258 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="34684" name="Firefox-127.0b1-msix-SSL"><location id="144728" os="win">/firefox/releases/127.0b1/win32/multi/Firefox%20Setup%20127.0b1.msix</location><location id="144729" os="win64">/firefox/releases/127.0b1/win64/multi/Firefox%20Setup%20127.0b1.msix</location><location id="144730" os="win64-aarch64">/firefox/releases/127.0b1/win64-aarch64/multi/Firefox%20Setup%20127.0b1.msix</location></product></locations>
2024-04-25 03:45:05,260 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '144728', 'path': '/firefox/releases/127.0b1/win32/multi/Firefox%20Setup%20127.0b1.msix'}, {'os': 'win64', 'id': '144729', 'path': '/firefox/releases/127.0b1/win64/multi/Firefox%20Setup%20127.0b1.msix'}, {'os': 'win64-aarch64', 'id': '144730', 'path': '/firefox/releases/127.0b1/win64-aarch64/multi/Firefox%20Setup%20127.0b1.msix'}]
2024-04-25 03:45:05,261 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/bouncerscript/script.py", line 166, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.9/site-packages/bouncerscript/script.py", line 70, in bouncer_submission
    check_locations_match(locations_paths, pr_config["paths_per_bouncer_platform"])
  File "/app/lib/python3.9/site-packages/bouncerscript/task.py", line 126, in check_locations_match
    raise ScriptWorkerTaskException("Bouncer entries are corrupt")
scriptworker.exceptions.ScriptWorkerTaskException: Bouncer entries are corrupt

Is that because of my previous staging release against D207087? If so, what can I do to "clean up" staging? (or should I just move on from 127.0b1?)

Comment 10

[Geoff Brown [:gbrown]]

Nevermind: Experiments confirmed that the error is associated with the 127.0b1 release and not caused by the current patch.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/83e1bb4d563f
Publish Windows aarch64 msix to archive.mozilla.org r=taskgraph-reviewers,bhearsum

Comment 12

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/83e1bb4d563f

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Geoff Brown [:gbrown] from comment #9)

I ran a staging release against a patch that only adds win64-aarch64 to beetmover (no bouncer changes) and release-bouncer-sub-firefox failed
https://treeherder.mozilla.org/jobs?repo=try&revision=cc2641cc74b575157c52b45cd603abc08c2a1cf3
https://treeherder.mozilla.org/logviewer?job_id=455765954&repo=try&lineNumber=432

2024-04-25 03:45:05,179 - bouncerscript.utils - INFO - Calling location_show?product=Firefox-127.0b1-msix-SSL with data: {}
2024-04-25 03:45:05,179 - bouncerscript.utils - INFO - Performing a GET request to https://dev.bounceradmin.nonprod.webservices.mozgcp.net/api/location_show?product=Firefox-127.0b1-msix-SSL with kwargs {'timeout': 60}
2024-04-25 03:45:05,258 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="34684" name="Firefox-127.0b1-msix-SSL"><location id="144728" os="win">/firefox/releases/127.0b1/win32/multi/Firefox%20Setup%20127.0b1.msix</location><location id="144729" os="win64">/firefox/releases/127.0b1/win64/multi/Firefox%20Setup%20127.0b1.msix</location><location id="144730" os="win64-aarch64">/firefox/releases/127.0b1/win64-aarch64/multi/Firefox%20Setup%20127.0b1.msix</location></product></locations>
2024-04-25 03:45:05,260 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '144728', 'path': '/firefox/releases/127.0b1/win32/multi/Firefox%20Setup%20127.0b1.msix'}, {'os': 'win64', 'id': '144729', 'path': '/firefox/releases/127.0b1/win64/multi/Firefox%20Setup%20127.0b1.msix'}, {'os': 'win64-aarch64', 'id': '144730', 'path': '/firefox/releases/127.0b1/win64-aarch64/multi/Firefox%20Setup%20127.0b1.msix'}]
2024-04-25 03:45:05,261 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/bouncerscript/script.py", line 166, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.9/site-packages/bouncerscript/script.py", line 70, in bouncer_submission
    check_locations_match(locations_paths, pr_config["paths_per_bouncer_platform"])
  File "/app/lib/python3.9/site-packages/bouncerscript/task.py", line 126, in check_locations_match
    raise ScriptWorkerTaskException("Bouncer entries are corrupt")
scriptworker.exceptions.ScriptWorkerTaskException: Bouncer entries are corrupt

Is that because of my previous staging release against D207087? If so, what can I do to "clean up" staging? (or should I just move on from 127.0b1?)

I have some thoughts in https://bugzilla.mozilla.org/show_bug.cgi?id=1893627#c1 but the tl;dr version is that bouncerscript is very strict about what it expects (to avoid accidents in production, I guess). There's some precedent for relaxing those restrictions on Try. In this particular case, I think it's fine to just let it fail until this work lands, and makes it into the simulations.

Comment 14

[Geoff Brown [:gbrown]]

Attachment: Bug 1890607 - Publish Windows aarch64 msix to archive.mozilla.org

Original Revision: https://phabricator.services.mozilla.com/D208539

Comment 15

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: aarch64 msix not copied to a.m.o; delays to aarch64 bouncer support
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: check that aarch64 msix is copied to a.m.o alongside win32 and win64
• Risk associated with taking this patch: low risk
• Explanation of risk level: just copies the aarch64 msix to a.m.o
• String changes made/needed: none
• Is Android affected?: no

Comment 16

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/e667bd7c0875

Comment 17

[Donal Meehan [:dmeehan]]

Backed out of beta, uplifted in error.
This has an open regression Bug 1893677

Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/48b546312eb48ce7b48e3ee20712d8041838c458

Comment 18

[Donal Meehan [:dmeehan]]

Comment on attachment 9398859 [details]
Bug 1890607 - Publish Windows aarch64 msix to archive.mozilla.org

Rejecting beta upllift request due to regressions in central linked to this patch

Comment 19

[Geoff Brown [:gbrown]]

Sorry for the confusion! Actually, that regression is not related to this bug. Can we go ahead with the beta uplift?

Comment 20

[Donal Meehan [:dmeehan]]

Sure, I can take this in a later push.

This bug is in a leave-open state, could this be resolved since a patch has landed in central?
Anything else needed could be tracked under a separate bug. It adds complexity to tracking otherwise.

Comment 21

[Geoff Brown [:gbrown]]

(In reply to Donal Meehan [:dmeehan] from comment #20)

This bug is in a leave-open state, could this be resolved since a patch has landed in central?
Anything else needed could be tracked under a separate bug. It adds complexity to tracking otherwise.

No problem - done.

Comment 22

[Geoff Brown [:gbrown]]

Nightly verified:

https://archive.mozilla.org/pub/firefox/nightly/2024/04/2024-04-26-09-19-52-mozilla-central/firefox-127.0a1.multi.win64-aarch64.installer.msix

Comment 23

[Nick Alexander :nalexander [he/him]]

(In reply to Geoff Brown [:gbrown] from comment #22)

Nightly verified:

https://archive.mozilla.org/pub/firefox/nightly/2024/04/2024-04-26-09-19-52-mozilla-central/firefox-127.0a1.multi.win64-aarch64.installer.msix

Amazing! Thank you, Geoff!

Comment 24

[Donal Meehan [:dmeehan]]

Comment on attachment 9398859 [details]
Bug 1890607 - Publish Windows aarch64 msix to archive.mozilla.org

Approved for 126.0b7

Comment 25

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/2886b76969c4

Comment 26

[Geoff Brown [:gbrown]]

https://archive.mozilla.org/pub/firefox/candidates/126.0b7-candidates/build1/win64-aarch64/multi/Firefox%20Setup%20126.0b7.msix