1934634 - Publish Linux/AArch64 builds for release, beta, and devedition

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement, P2)

Description

[Gabriel Bustamante [:gabriel]]

[Tracking Requested - why for this release]: I want to land this on the 134 Beta.

Comment 1

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Publish linux64-aarch64 builds on release branches r=#releng-reviewers

Comment 2

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add bouncer submissions for linux64-aarch64 r=#releng-reviewers

Comment 3

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Run update verify for linux64-aarch64 r=#releng-reviewers

Comment 4

[Gabriel Bustamante [:gabriel]]

Attachment: WIP: Bug 1934634 - Configure bouncer checks and update verification tasks for devedition Linux/AArch64 builds r=#releng-reviewers

Comment 5

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 bouncer checks for latest beta and devedition r=#releng-reviewers

Comment 6

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 bouncer checks for latest release r=#releng-reviewers

Comment 7

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Update release-update-verify to include release, beta, and devedition AArch64 builds r=#releng-reviewers

Comment 8

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Configure bouncer checks for release builds targeting Linux/AArch64 r=#releng-reviewers

Comment 9

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Run shippable build tasks for devedition on linux64-aarch64 r=#releng-reviewers

Comment 10

[Gabriel Bustamante [:gabriel]]

Attachment: WIP: Bug 1934634 - Upload signed langpack addons for linux-aarch64 releases to the Mozilla archive r=#releng-reviewers

Comment 11

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add beta/devedition linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Comment 12

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add beta/devedition linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 13

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Generate locale changesets in l10n-bumper for the linux64-aarch64-devedition platform r=#releng-reviewers

Comment 14

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64-devedition platform to the merge-automation action r=#releng-reviewers

Comment 15

[Gabriel Bustamante [:gabriel]]

Attachment: Add linux aarch64 support in scriptworker-scripts

Comment 16

[[e-mail] (:bhearsum)]

I had a fairly exhaustive look through the Taskcluster directory by applying your stack and looking through all files with linux64 in them.

In addition to the patches you have and comments I left there, I think you'll need additional changes:

• To add support in partner repack kinds, eg: https://searchfox.org/mozilla-central/source/taskcluster/kinds/release-partner-repack/kind.yml (but there are others as well) as well as partner related transforms and utils (eg: https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/util/partners.py)
• target_tasks.py needs an update

And while it doesn't block this bug, please get something on file to add startup tests for linux64-aarch64 and support for update-verify-next (which we may need win64-aarch64 support for as well?).

Comment 17

[[e-mail] (:bhearsum)]

Julien - can you take a look at this stack as well?

Comment 18

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 to filter_release_tasks function in target_tasks r=#releng-reviewers

Comment 19

[Gabriel Bustamante [:gabriel]]

I opened a bug about the partner repacks.

https://bugzilla.mozilla.org/show_bug.cgi?id=1936358

Should we hold off on publishing Linux/AArch64 builds for release, beta, and devedition?

I started looking at these, but it's still a WIP and I'd like to run them on staging.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Gabriel Bustamante [:gabriel] from comment #19)

I opened a bug about the partner repacks.

https://bugzilla.mozilla.org/show_bug.cgi?id=1936358

Should we hold off on publishing Linux/AArch64 builds for release, beta, and devedition?

I started looking at these, but it's still a WIP and I'd like to run them on staging.

Yeah, we need to block on this. We have at least two partner repacks on Linux; we can't reasonably ship without them AFAIK. (We could probably live without them for the early part of the Beta cycle, but I think it would be best to avoid hitting Beta until we're certain partner repacks will be ready in time for Release.)

Comment 21

[Julien Cristau [:jcristau]]

Looks good to me for the most part, left a couple of comments in phab.

Comment 22

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux aarch64 devedition platform to on-change l10n changesets r=#releng-reviewers

Comment 23

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - raise an ValueError in the l10n transform if there are not l10n changesets for a platform in the locales file r=#releng-reviewers

Comment 24

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Comment 25

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 26

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add esr linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Comment 27

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add esr linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 28

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux aarch64 devedition platform to on-change l10n changesets r=#releng-reviewers

Comment 29

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - raise an ValueError in the l10n transform if there are not l10n changesets for a platform in the locales file r=#releng-reviewers

Comment 30

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Comment 31

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 32

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add esr linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Comment 33

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 bouncer checks for latest esr r=#releng-reviewers

Comment 34

[Gabriel Bustamante [:gabriel]]

Attachment: WIP: Bug 1934634 - Run update verify for linux64-aarch64 r=#releng-reviewers

Comment 35

[Gabriel Bustamante [:gabriel]]

Attachment: WIP: Bug 1934634 - Add beta/devedition linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 36

[Gabriel Bustamante [:gabriel]]

Attachment: WIP: Bug 1934634 - Add linux64-aarch64 bouncer checks for latest beta and devedition r=#releng-reviewers

Comment 37

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Run update verify for linux64-aarch64 r=#releng-reviewers

Comment 38

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Comment 39

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 bouncer checks for latest release r=#releng-reviewers

Comment 40

[Gabriel Bustamante [:gabriel]]

I should probably leave out some of ESR patches in this stack (namely D235869, D235677)

...I might open a "Publish Linux/AArch64 builds for esr <next ESR>" bug.

D235676 can land with the rest of them on central (to run bouncer check on the ESR completes, langpacks, etc, on the first ESR release with Linux/ARM64 builds.) But iiuc we will have to uplift UV and bouncer checks for latest/partials to the next ESR branch after the first ESR release with Linux/ARM64 builds. Then we can land the ESR ARM64 patches for those things on central.

Comment 41

[Phabricator Automation]

Comment on attachment 9462022 [details]
Bug 1934634 - Add esr linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=#releng-reviewers

Revision D235676 was moved to bug 1944440. Setting attachment 9462022 [details] to obsolete.

Comment 42

[Phabricator Automation]

Comment on attachment 9462023 [details]
Bug 1934634 - Add esr linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Revision D235677 was moved to bug 1944440. Setting attachment 9462023 [details] to obsolete.

Comment 43

[Phabricator Automation]

Comment on attachment 9462343 [details]
Bug 1934634 - Add linux64-aarch64 bouncer checks for latest esr r=#releng-reviewers

Revision D235869 was moved to bug 1944440. Setting attachment 9462343 [details] to obsolete.

Comment 44

[Donal Meehan [:dmeehan]]

:gabriel a beta uplift request was added here, likely because the patch was updated.
Just double checking before I remove it, so it doesn't show in our dashboard

Comment 45

[Gabriel Bustamante [:gabriel]]

Woops, you are correct, I went ahead and removed it. Excuse the noise :)

Comment 46

[Gabriel Bustamante [:gabriel]]

Looking pretty good in staging now.

Here's my most recent staging release:

https://treeherder.mozilla.org/jobs?repo=try&revision=60d58ec72d68ee336da62492eb0ea5ce489e12b1

Got some failures on the OSX jobs because of Bug 1945900

I was able to run the bouncer check with partials/latest releases, and UV looks good on Linux/ARM64.

We should consider uplifting this to, perhaps, b5? The patches won't do anything on central.

Comment 47

[Gabriel Bustamante [:gabriel]]

wow :/ the phab bot added the approval-mozilla-beta? flag even tho I marked them as changes planned...

Anyways...

Here's a staging release with the latest version of those beta patches: https://treeherder.mozilla.org/jobs?repo=try&revision=c045b4d696bc96829cd19ddafee7776a8c5092fe

Comment 48

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Generate locale changesets in l10n-bumper for the linux64-aarch64-devedition platform

Original Revision: https://phabricator.services.mozilla.com/D233274

Comment 49

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux aarch64 devedition platform to on-change l10n changesets

Original Revision: https://phabricator.services.mozilla.com/D235691

Comment 50

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Run shippable build tasks for devedition on linux64-aarch64

Original Revision: https://phabricator.services.mozilla.com/D232003

Comment 51

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Publish linux64-aarch64 builds on release branches

Original Revision: https://phabricator.services.mozilla.com/D230803

Comment 52

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64-devedition platform to the merge-automation action

Original Revision: https://phabricator.services.mozilla.com/D233275

Comment 53

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add linux64-aarch64 to filter_release_tasks function in target_tasks

Original Revision: https://phabricator.services.mozilla.com/D234213

Comment 54

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add bouncer submissions for linux64-aarch64

Original Revision: https://phabricator.services.mozilla.com/D231435

Comment 55

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add beta/devedition linux64-aarch64 installers, langpacks, and completes to bouncer check configuration

Original Revision: https://phabricator.services.mozilla.com/D232552

Comment 56

[Gabriel Bustamante [:gabriel]]

Attachment: Bug 1934634 - Add release linux64-aarch64 installers, langpacks, and completes to bouncer check configuration

Original Revision: https://phabricator.services.mozilla.com/D235674

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fcc2592b20ca
Generate locale changesets in l10n-bumper for the linux64-aarch64-devedition platform r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/7a066a9e6815
Add linux aarch64 devedition platform to on-change l10n changesets r=releng-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/d21d20a6840b
Run shippable build tasks for devedition on linux64-aarch64 r=releng-reviewers,bhearsum,jcristau
https://hg.mozilla.org/integration/autoland/rev/deaca566d4bb
Publish linux64-aarch64 builds on release branches r=releng-reviewers,taskgraph-reviewers,bhearsum,jcristau
https://hg.mozilla.org/integration/autoland/rev/0297ea965ca5
Add linux64-aarch64-devedition platform to the merge-automation action r=releng-reviewers,bhearsum,jcristau,taskgraph-reviewers
https://hg.mozilla.org/integration/autoland/rev/fac110ca38f3
Add linux64-aarch64 to filter_release_tasks function in target_tasks r=releng-reviewers,taskgraph-reviewers,bhearsum,jcristau
https://hg.mozilla.org/integration/autoland/rev/447a66411dd4
Add bouncer submissions for linux64-aarch64 r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/d13d2884441d
Add beta/devedition linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/ee976bb893e9
Add release linux64-aarch64 installers, langpacks, and completes to bouncer check configuration r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 58

[amarc]

https://hg.mozilla.org/mozilla-central/rev/fcc2592b20ca
https://hg.mozilla.org/mozilla-central/rev/7a066a9e6815
https://hg.mozilla.org/mozilla-central/rev/d21d20a6840b
https://hg.mozilla.org/mozilla-central/rev/deaca566d4bb
https://hg.mozilla.org/mozilla-central/rev/0297ea965ca5
https://hg.mozilla.org/mozilla-central/rev/fac110ca38f3
https://hg.mozilla.org/mozilla-central/rev/447a66411dd4
https://hg.mozilla.org/mozilla-central/rev/d13d2884441d
https://hg.mozilla.org/mozilla-central/rev/ee976bb893e9

Comment 59

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/080f5c90fa92
https://hg.mozilla.org/releases/mozilla-beta/rev/6bb9fc13aac4
https://hg.mozilla.org/releases/mozilla-beta/rev/4e3cefeff39b
https://hg.mozilla.org/releases/mozilla-beta/rev/5d573391cc69
https://hg.mozilla.org/releases/mozilla-beta/rev/363b5702181a
https://hg.mozilla.org/releases/mozilla-beta/rev/361534964f51
https://hg.mozilla.org/releases/mozilla-beta/rev/0385ae58fd0c
https://hg.mozilla.org/releases/mozilla-beta/rev/41efdd71af44
https://hg.mozilla.org/releases/mozilla-beta/rev/5e8c6c43b8af

Comment 60

[Donal Meehan [:dmeehan]]

Mentioned in Slack but adding a comment here also
This is in a leave-open state. Could the patches pending please be moved to a different bug?
It will be difficult to track across releases when those patches land later under the same bug.

Comment 61

[Phabricator Automation]

Comment on attachment 9442227 [details]
Bug 1934634 - Run update verify for linux64-aarch64 r=#releng-reviewers

Revision D231436 was moved to bug 1947847. Setting attachment 9442227 [details] to obsolete.

Comment 62

[Phabricator Automation]

Comment on attachment 9444444 [details]
Bug 1934634 - Add beta/devedition linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Revision D232553 was moved to bug 1947848. Setting attachment 9444444 [details] to obsolete.

Comment 63

[Phabricator Automation]

Comment on attachment 9442229 [details]
Bug 1934634 - Add linux64-aarch64 bouncer checks for latest beta and devedition r=#releng-reviewers

Revision D231438 was moved to bug 1947848. Setting attachment 9442229 [details] to obsolete.

Comment 64

[Phabricator Automation]

Comment on attachment 9462021 [details]
Bug 1934634 - Add release linux64-aarch64 partials to bouncer check configuration r=#releng-reviewers

Revision D235675 was moved to bug 1947848. Setting attachment 9462021 [details] to obsolete.

Comment 65

[Phabricator Automation]

Comment on attachment 9442230 [details]
Bug 1934634 - Add linux64-aarch64 bouncer checks for latest release r=#releng-reviewers

Revision D231439 was moved to bug 1947848. Setting attachment 9442230 [details] to obsolete.

Comment 66

[Phabricator Automation]

Comment on attachment 9462047 [details]
Bug 1934634 - raise an ValueError in the l10n transform if there are not l10n changesets for a platform in the locales file r=#releng-reviewers

Revision D235692 was moved to bug 1947857. Setting attachment 9462047 [details] to obsolete.

Comment 67

[Julien Cristau [:jcristau]]

bouncer-check changes backed out of beta for breaking cron-bouncer-check. They need to wait until after the first release on each branch.

https://hg.mozilla.org/releases/mozilla-beta/rev/48a4e12b1e8b875a928190cbcb6633178bfbb419

Comment 68

[Gabriel Bustamante [:gabriel]]

Release Note Request (optional, but appreciated)
[Why is this notable]: We are adding Firefox for Desktop support for a new platform (Linux/ARM64)
[Affects Firefox for Android]: No
[Suggested wording]: Mozilla Firefox for Linux is now available on ARM64 (AArch64), with installation options via APT and tarballs. Flatpak support is coming soon!
[Links (documentation, blog post, etc)]: https://support.mozilla.org/en-US/kb/install-firefox-linux#w_install-firefox-deb-package-for-debian-based-distributions-recommended