Release Fennec Nightly AArch64 APK to Google Play

RESOLVED FIXED

Status

enhancement
P1
normal
RESOLVED FIXED
2 years ago
3 months ago

People

(Reporter: jlorenzo, Assigned: jlorenzo)

Tracking

(Depends on 1 bug)

unspecified
ARM64
Android
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 fixed, firefox55 fixed, firefox64 wontfix, firefox65 wontfix, firefox66 fixed)

Details

(Whiteboard: [arm64:m1])

Attachments

(6 attachments)

Assignee

Description

2 years ago
Bug 1366404 reached master, making the pushapk job fail[1]. In this bug, I'm going to:

1. blacklist AArch64 from pushapk, so we can resume publication on Google Play Store.
2. Ask confirmation that we do want AArch64 on Google Play Store, as soon as possible
3. Once AArch is mature enough, support this new arch in Google Play.

[1] https://public-artifacts.taskcluster.net/UoHM4CBtQEavVLn-0qNqEw/0/public/logs/live_backing.log
Comment hidden (mozreview-request)
Assignee

Updated

2 years ago
Assignee: nobody → jlorenzo

Comment 2

2 years ago
mozreview-review
Comment on attachment 8872357 [details]
Bug 1368484 - Don't upload AArch64 APK to Google Play

https://reviewboard.mozilla.org/r/143840/#review147590
Attachment #8872357 - Flags: review?(bhearsum) → review+

Comment 3

2 years ago
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d8c239da0598
Don't upload AArch64 APK to Google Play r=bhearsum
Assignee

Comment 4

2 years ago
AArch64 will be deactivated from push-apk once attachment 8872357 [details] reaches master.

Question 2 has been answered by :snop on IRC. TL;DR: We don't want to ship aarch64 yet. 

> 16:36:38 <snorp> jlorenzo: we do not want to ship it
> 16:36:41 <snorp> jlorenzo: except maybe Nightly
> 16:36:45 <snorp> when nightly goes to Play
> 16:38:24 <jlorenzo> snorp: okay. Does "maybe Nightly" mean "I would prefer to have real users on it asap"? Or "I'd like to hide it from the user base and have QE test it first"? 
> 16:38:57 <snorp> jlorenzo: the latter
> 16:39:18 <snorp> jlorenzo: if we put Nightly up it should be under a different package name, for instance
> 16:39:26 <snorp> Nightly64 or something
> 16:39:56 <jlorenzo> snorp: great! What's the risk to ship it alongside regular nightly? 
> 16:40:19 <jlorenzo> (release management would like to ship nightly tomorrow, if I'm not mistaken) 
> 16:40:26 <snorp> jlorenzo: 64-bit devices would always get the new 64-bit version, and which would be bad because we're not shipping that :)
> 16:42:34 <jlorenzo> snorp: that sounds correct to me. Moreover, there is no way to unpublish an APK on GOogle Play. This means, 64bit-devices will be stuck on aarch64 forever 
16:42:48 <snorp> jlorenzo: right, so let's not do that.

Comment 5

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d8c239da0598
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Assignee

Comment 6

2 years ago
Oops, I forgot the leave-open keyword
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---

Updated

2 years ago
Priority: -- → P1
Duplicate of this bug: 1438080
Component: Release Automation: Other → Release Automation: Pushapk
Assignee

Updated

11 months ago
Depends on: 1477259

Comment 8

11 months ago
FYI, I had accidentally installed an aarch64 APK on my tablet (Snapdragon 820) and noticed that it's significantly slower.  Some simple benchmarks like sunspider and speed-battle.com show it to be 2-9x slower than the arm32 build.

So, some performance testing should be done before this is pushed to Google Play..

Comment 9

11 months ago
(In reply to robsmith11 from comment #8)
> FYI, I had accidentally installed an aarch64 APK on my tablet (Snapdragon
> 820) and noticed that it's significantly slower.  Some simple benchmarks
> like sunspider and speed-battle.com show it to be 2-9x slower than the arm32
> build.

We currently have only non-optimizing JS and WebAssembly compilers for aarch64, which is probably the main reason you're seeing this.
Adding a dependency on ARM64 Ion meta bug 1380129 because we don't want to publish ARM64 builds of Fennec to the Google Play Store without ARM64 Ion.
Depends on: arm64-ion
No longer blocks: 1366404
Depends on: Fennec-ARM64, 1366404
No longer depends on: arm64-ion
Summary: Release Nighly AArch64 APK to Google Play → Release Fennec Nightly AArch64 APK to Google Play
Whiteboard: [arm64:m2]
Assignee

Comment 11

6 months ago
For reference, shipping AArch64 should be a matter of 

* reverting attachment 8872357 [details]. This patch was about changing the task definition to exclude AArch64. We should put it back in.
* modifying mozapkpublisher to accept AArch64 starting a given Firefox version. This is done by referencing the architecture in [1] and adding a new test case at [2].
* releasing a new version of mozapkpublisher and deploying it to build-puppet[3] so the worker takes the changes into account.

Please note, there are some plans to changes the [1] and [2] pointers. See the discussion at [4]. I don't think the complexity will evolve, though.


That said, allowing a new architecture to be shipped is easy enough: about a day of work (including tests and deployment).

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".



[1] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/common/apk/history.py#L24-L32
[2] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/test/common/apk/test_history.py#L21-L26
[3] https://github.com/mozilla-releng/build-puppet
[4] https://github.com/mozilla-releng/pushapkscript/issues/65

(In reply to Johan Lorenzo [:jlorenzo] from comment #11)

That said, allowing a new architecture to be shipped is easy enough: about a
day of work (including tests and deployment).

I see a problem though: If AArch64 doesn't ride the trains - as in "we
shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as
Firefox Beta 65" - we need to revisit the logic or amend the configuration
to say "AArch actually started to shipped in Firefox 66".

Johan, since adding a new architecture is probably a rare event and AArch64 will only need to skip one or two Nightly->Beta merges, could we just manually revert the AArch64 task changes in the Beta channel after each merge?

btw, head ups: this week I will probably submit the bug to request we ship AARch64 Fennec Nightly. I need to confirm some Android test issues first.

Flags: needinfo?(jlorenzo)

I was about file a new bug, but this bug is still open so we can use it.

We are now running (bug 1501584) and passing tests for ARM64 Fennec on mozilla-central, autoland, and inbound on real ARM64 devices (Google Pixel 2).

We would like to release ARM64 Fennec 66 to the Nightly channel, but not ride the trains to Beta yet. There are still ARM64 bugs we need to fix before riding the trains:

https://wiki.mozilla.org/Mobile/ARM64

Whiteboard: [arm64:m2] → [arm64:m1]
Assignee

Comment 14

6 months ago

(In reply to Chris Peterson [:cpeterson] from comment #12)

could we just manually revert the AArch64 task changes in the Beta channel after each merge?

I'm sorry. I just read my previous comment again, and I realize this part:

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".

is not really clear. Basically, if we want to "revert" AArch64 from beta, we'd have to do almost the same 3 steps as mentioned in comment 11. It should be around a day of work, the hardest part is to remember to do it.

Flags: needinfo?(jlorenzo)

(In reply to Johan Lorenzo [:jlorenzo] from comment #14)

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".

is not really clear. Basically, if we want to "revert" AArch64 from beta, we'd have to do almost the same 3 steps as mentioned in comment 11. It should be around a day of work, the hardest part is to remember to do it.

So you could add AArch64 to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE in mozapkpublisher [1] and then, after 65 Nightly rides to Beta, just remove the AArch64 line from _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE?

Is it possible to write the code so that AArch64 is only added to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE if channel == nightly? It would be hacky, but then you wouldn't need to spend a second or third day reverting AArch64 from the Beta channel. Once AArch64 is ready to ride the trains, you could remove the conditional hack.

The Fennec team like to start publishing AArch64 Fennec Nightly whenever you or another release engineer can schedule the day of work.

[1] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/common/apk/history.py#L24-L32

Flags: needinfo?(jlorenzo)
Assignee

Comment 16

5 months ago
Release Fennec Nightly (only) AArch64 APK to Google Play
Assignee

Comment 17

5 months ago

r+'d by mtabara at: https://github.com/mozilla-releng/mozapkpublisher/pull/152#pullrequestreview-192117837
Landed on master at: https://github.com/mozilla-releng/mozapkpublisher/commit/36444cc8ad5b041b2752665df5bec7edcfc56c14

(In reply to Chris Peterson [:cpeterson] from comment #15)

Is it possible to write the code so that AArch64 is only added to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE if channel == nightly? It would be hacky, but then you wouldn't need to spend a second or third day reverting AArch64 from the Beta channel. Once AArch64 is ready to ride the trains, you could remove the conditional hack.

Good idea! I implemented the logic this way. Bonus: We won't have to revert what's in-tree either, thanks to the same logic.

The Fennec team like to start publishing AArch64 Fennec Nightly whenever you or another release engineer can schedule the day of work.

I think I can land this today, so the first AArch64 Nightly would be published tomorrow.

Flags: needinfo?(jlorenzo)
Attachment #9036329 - Flags: review+
Assignee

Comment 18

5 months ago

Comment on attachment 9036329 [details] [review]
[mozapkpublisher] push-apk: Allow AArch64 on Nightly only

See previous comment.

Attachment #9036329 - Flags: checked-in+

Comment 20

5 months ago
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5306b81a94f7
Release Fennec Nightly (only) AArch64 APK to Google Play r=mtabara

Loading the following test page in Fennec will confirm whether you have ARMv7 (32-bit) or ARM64 Fennec installed:

https://is.gd/navigatorplatform -> https://codepen.io/anon/pen/JwxbYQ

Assignee

Comment 26

5 months ago

The very first AArch64 APK has been uploaded to the Google Play store: https://tools.taskcluster.net/groups/frz3aHJdTGOgKx7xCDKkmw/tasks/COm1E5W_Tc2K4Cerkorg6Q/runs/2/logs/public%2Flogs%2Flive_backing.log#L206. I can see on the artifact library of GP.

We're done with it, then. Please note it won't ride the trains, unless someone amend the patches attached to this bug. There is no action to take when 66 (or above) reaches beta.

Status: REOPENED → RESOLVED
Closed: 2 years ago5 months ago
Resolution: --- → FIXED
Depends on: 1520261

(In reply to Johan Lorenzo [:jlorenzo] from comment #26)

The very first AArch64 APK has been uploaded to the Google Play store: https://tools.taskcluster.net/groups/frz3aHJdTGOgKx7xCDKkmw/tasks/COm1E5W_Tc2K4Cerkorg6Q/runs/2/logs/public%2Flogs%2Flive_backing.log#L206. I can see on the artifact library of GP.

Thanks, Johan!

Looks like a few ARM64 Fennec crash reports are already starting to roll in:

https://crash-stats.mozilla.com/search/?cpu_arch=arm64&build_id=%3E%3D20190114000000&product=FennecAndroid&version=66.0a1&date=%3E%3D2019-01-13T11%3A18%3A00.000Z&date=%3C2019-01-30T19%3A18%3A00.000Z&_sort=-date&_facets=signature&_facets=build_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

We're done with it, then. Please note it won't ride the trains, unless someone amend the patches attached to this bug. There is no action to take when 66 (or above) reaches beta.

I filed bug 1520261 to track the Releng work when ARM64 Fennec is ready to ride the trains.

Comment hidden (Intermittent Failures Robot)
See Also: → Fennec-x86_64
Keywords: leave-open
OS: Unspecified → Android
Hardware: Unspecified → ARM64

(In reply to Tom Prince [:tomprince] from comment #29)

https://hg.mozilla.org/releases/mozilla-esr60/rev/b01dbd843889

Tom, why did these Android changes need to be uplifted to ESR 60? ARM64 Fennec Nightly was enabled in 66, not 60, and we don't publish any ESR Fennec builds to the Google Play Store anyway.

Flags: needinfo?(mozilla)

They probably didn't strictly need to be uplifted. Each release, I've been uplifting changes to taskgraph and release related code to esr60 (see here). This patch got uplifted, as it makes it easier to look at the diff between esr and m-c, and since, as you mentioned, we don't build fennec on ESR, it is particularly low risk.

Flags: needinfo?(mozilla)

(In reply to Tom Prince [:tomprince] from comment #31)

They probably didn't strictly need to be uplifted. Each release, I've been uplifting changes to taskgraph and release related code to esr60 (see here). This patch got uplifted, as it makes it easier to look at the diff between esr and m-c, and since, as you mentioned, we don't build fennec on ESR, it is particularly low risk.

Sounds good. I just wanted to make sure there wasn't some AArch64 Fennec problem that I hadn't heard about yet. :)

Assignee

Updated

3 months ago
Blocks: 1540152
You need to log in before you can comment on or make changes to this bug.