1368484 - Release Fennec Nightly AArch64 APK to Google Play

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P1)

Description

[Johan Lorenzo [:jlorenzo]]

Bug 1366404 reached master, making the pushapk job fail[1]. In this bug, I'm going to:

1. blacklist AArch64 from pushapk, so we can resume publication on Google Play Store.
2. Ask confirmation that we do want AArch64 on Google Play Store, as soon as possible
3. Once AArch is mature enough, support this new arch in Google Play.

[1] https://public-artifacts.taskcluster.net/UoHM4CBtQEavVLn-0qNqEw/0/public/logs/live_backing.log

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1368484 - Don't upload AArch64 APK to Google Play

Review commit: https://reviewboard.mozilla.org/r/143840/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/143840/

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 8872357 [details]
Bug 1368484 - Don't upload AArch64 APK to Google Play

https://reviewboard.mozilla.org/r/143840/#review147590

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d8c239da0598
Don't upload AArch64 APK to Google Play r=bhearsum

Comment 4

[Johan Lorenzo [:jlorenzo]]

AArch64 will be deactivated from push-apk once attachment 8872357 [details] reaches master.

Question 2 has been answered by :snop on IRC. TL;DR: We don't want to ship aarch64 yet. 

> 16:36:38 <snorp> jlorenzo: we do not want to ship it
> 16:36:41 <snorp> jlorenzo: except maybe Nightly
> 16:36:45 <snorp> when nightly goes to Play
> 16:38:24 <jlorenzo> snorp: okay. Does "maybe Nightly" mean "I would prefer to have real users on it asap"? Or "I'd like to hide it from the user base and have QE test it first"? 
> 16:38:57 <snorp> jlorenzo: the latter
> 16:39:18 <snorp> jlorenzo: if we put Nightly up it should be under a different package name, for instance
> 16:39:26 <snorp> Nightly64 or something
> 16:39:56 <jlorenzo> snorp: great! What's the risk to ship it alongside regular nightly? 
> 16:40:19 <jlorenzo> (release management would like to ship nightly tomorrow, if I'm not mistaken) 
> 16:40:26 <snorp> jlorenzo: 64-bit devices would always get the new 64-bit version, and which would be bad because we're not shipping that :)
> 16:42:34 <jlorenzo> snorp: that sounds correct to me. Moreover, there is no way to unpublish an APK on GOogle Play. This means, 64bit-devices will be stuck on aarch64 forever 
16:42:48 <snorp> jlorenzo: right, so let's not do that.

Comment 5

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d8c239da0598

Comment 6

[Johan Lorenzo [:jlorenzo]]

Oops, I forgot the leave-open keyword

Comment 8

[robsmith11]

FYI, I had accidentally installed an aarch64 APK on my tablet (Snapdragon 820) and noticed that it's significantly slower.  Some simple benchmarks like sunspider and speed-battle.com show it to be 2-9x slower than the arm32 build.

So, some performance testing should be done before this is pushed to Google Play..

Comment 9

[Lars T Hansen [:lth]]

(In reply to robsmith11 from comment #8)
> FYI, I had accidentally installed an aarch64 APK on my tablet (Snapdragon
> 820) and noticed that it's significantly slower.  Some simple benchmarks
> like sunspider and speed-battle.com show it to be 2-9x slower than the arm32
> build.

We currently have only non-optimizing JS and WebAssembly compilers for aarch64, which is probably the main reason you're seeing this.

Comment 10

[Chris Peterson [:cpeterson]]

Adding a dependency on ARM64 Ion meta bug 1380129 because we don't want to publish ARM64 builds of Fennec to the Google Play Store without ARM64 Ion.

Comment 11

[Johan Lorenzo [:jlorenzo]]

For reference, shipping AArch64 should be a matter of 

* reverting attachment 8872357 [details]. This patch was about changing the task definition to exclude AArch64. We should put it back in.
* modifying mozapkpublisher to accept AArch64 starting a given Firefox version. This is done by referencing the architecture in [1] and adding a new test case at [2].
* releasing a new version of mozapkpublisher and deploying it to build-puppet[3] so the worker takes the changes into account.

Please note, there are some plans to changes the [1] and [2] pointers. See the discussion at [4]. I don't think the complexity will evolve, though.


That said, allowing a new architecture to be shipped is easy enough: about a day of work (including tests and deployment).

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".



[1] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/common/apk/history.py#L24-L32
[2] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/test/common/apk/test_history.py#L21-L26
[3] https://github.com/mozilla-releng/build-puppet
[4] https://github.com/mozilla-releng/pushapkscript/issues/65

Comment 12

[Chris Peterson [:cpeterson]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #11)

That said, allowing a new architecture to be shipped is easy enough: about a
day of work (including tests and deployment).

I see a problem though: If AArch64 doesn't ride the trains - as in "we
shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as
Firefox Beta 65" - we need to revisit the logic or amend the configuration
to say "AArch actually started to shipped in Firefox 66".

Johan, since adding a new architecture is probably a rare event and AArch64 will only need to skip one or two Nightly->Beta merges, could we just manually revert the AArch64 task changes in the Beta channel after each merge?

btw, head ups: this week I will probably submit the bug to request we ship AARch64 Fennec Nightly. I need to confirm some Android test issues first.

Comment 13

[Chris Peterson [:cpeterson]]

I was about file a new bug, but this bug is still open so we can use it.

We are now running (bug 1501584) and passing tests for ARM64 Fennec on mozilla-central, autoland, and inbound on real ARM64 devices (Google Pixel 2).

We would like to release ARM64 Fennec 66 to the Nightly channel, but not ride the trains to Beta yet. There are still ARM64 bugs we need to fix before riding the trains:

https://wiki.mozilla.org/Mobile/ARM64

Comment 14

[Johan Lorenzo [:jlorenzo]]

(In reply to Chris Peterson [:cpeterson] from comment #12)

could we just manually revert the AArch64 task changes in the Beta channel after each merge?

I'm sorry. I just read my previous comment again, and I realize this part:

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".

is not really clear. Basically, if we want to "revert" AArch64 from beta, we'd have to do almost the same 3 steps as mentioned in comment 11. It should be around a day of work, the hardest part is to remember to do it.

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #14)

I see a problem though: If AArch64 doesn't ride the trains - as in "we shipped AArch64 in Firefox Nightly 65, but we don't want to ship it as Firefox Beta 65" - we need to revisit the logic or amend the configuration to say "AArch actually started to shipped in Firefox 66".

is not really clear. Basically, if we want to "revert" AArch64 from beta, we'd have to do almost the same 3 steps as mentioned in comment 11. It should be around a day of work, the hardest part is to remember to do it.

So you could add AArch64 to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE in mozapkpublisher [1] and then, after 65 Nightly rides to Beta, just remove the AArch64 line from _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE?

Is it possible to write the code so that AArch64 is only added to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE if channel == nightly? It would be hacky, but then you wouldn't need to spend a second or third day reverting AArch64 from the Beta channel. Once AArch64 is ready to ride the trains, you could remove the conditional hack.

The Fennec team like to start publishing AArch64 Fennec Nightly whenever you or another release engineer can schedule the day of work.

[1] https://github.com/mozilla-releng/mozapkpublisher/blob/b6706efa7f759a60feec456e74a12fa36bf4c1ce/mozapkpublisher/common/apk/history.py#L24-L32

Comment 16

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1368484 - Release Fennec Nightly (only) AArch64 APK to Google Play r=mtabara

Release Fennec Nightly (only) AArch64 APK to Google Play

Comment 17

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozapkpublisher] push-apk: Allow AArch64 on Nightly only

r+'d by mtabara at: https://github.com/mozilla-releng/mozapkpublisher/pull/152#pullrequestreview-192117837
Landed on master at: https://github.com/mozilla-releng/mozapkpublisher/commit/36444cc8ad5b041b2752665df5bec7edcfc56c14

(In reply to Chris Peterson [:cpeterson] from comment #15)

Is it possible to write the code so that AArch64 is only added to _MAJOR_FIREFOX_VERSIONS_PER_ARCHITECTURE if channel == nightly? It would be hacky, but then you wouldn't need to spend a second or third day reverting AArch64 from the Beta channel. Once AArch64 is ready to ride the trains, you could remove the conditional hack.

Good idea! I implemented the logic this way. Bonus: We won't have to revert what's in-tree either, thanks to the same logic.

The Fennec team like to start publishing AArch64 Fennec Nightly whenever you or another release engineer can schedule the day of work.

I think I can land this today, so the first AArch64 Nightly would be published tomorrow.

Comment 18

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9036329 [details] [review]
[mozapkpublisher] push-apk: Allow AArch64 on Nightly only

See previous comment.

Comment 19

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Bump mozapkpublisher to 0.12.0

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5306b81a94f7
Release Fennec Nightly (only) AArch64 APK to Google Play r=mtabara

Comment 21

[Chris Peterson [:cpeterson]]

Loading the following test page in Fennec will confirm whether you have ARMv7 (32-bit) or ARM64 Fennec installed:

https://is.gd/navigatorplatform -> https://codepen.io/anon/pen/JwxbYQ

Comment 22

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/5306b81a94f7

Comment 23

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozapkpublisher] Allow AArch64 to have a different API level

r+'d by mtabara at https://github.com/mozilla-releng/mozapkpublisher/pull/153#pullrequestreview-192687337
Landed on master at https://github.com/mozilla-releng/mozapkpublisher/commit/7aa2bcef21e8ea1d943c5031681abe9ac7f7d10d
Released as mozapkpublisher 0.13.0: https://pypi.org/project/mozapkpublisher/0.13.0/ https://github.com/mozilla-releng/mozapkpublisher/releases/tag/0.13.0

Comment 24

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Bump mozapkpublisher to 0.13.0

Comment 25

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9036611 [details] [review]
[build/puppet] Bump mozapkpublisher to 0.13.0

r+'d by mtabara at https://github.com/mozilla-releng/build-puppet/pull/364#pullrequestreview-192715534
Landed on master at https://github.com/mozilla-releng/build-puppet/commit/0a3342f430ba1fa585de9a2d5f4852069b63c519

Comment 26

[Johan Lorenzo [:jlorenzo]]

The very first AArch64 APK has been uploaded to the Google Play store: https://tools.taskcluster.net/groups/frz3aHJdTGOgKx7xCDKkmw/tasks/COm1E5W_Tc2K4Cerkorg6Q/runs/2/logs/public%2Flogs%2Flive_backing.log#L206. I can see on the artifact library of GP.

We're done with it, then. Please note it won't ride the trains, unless someone amend the patches attached to this bug. There is no action to take when 66 (or above) reaches beta.

Comment 27

[Chris Peterson [:cpeterson]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #26)

The very first AArch64 APK has been uploaded to the Google Play store: https://tools.taskcluster.net/groups/frz3aHJdTGOgKx7xCDKkmw/tasks/COm1E5W_Tc2K4Cerkorg6Q/runs/2/logs/public%2Flogs%2Flive_backing.log#L206. I can see on the artifact library of GP.

Thanks, Johan!

Looks like a few ARM64 Fennec crash reports are already starting to roll in:

https://crash-stats.mozilla.com/search/?cpu_arch=arm64&build_id=%3E%3D20190114000000&product=FennecAndroid&version=66.0a1&date=%3E%3D2019-01-13T11%3A18%3A00.000Z&date=%3C2019-01-30T19%3A18%3A00.000Z&_sort=-date&_facets=signature&_facets=build_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

We're done with it, then. Please note it won't ride the trains, unless someone amend the patches attached to this bug. There is no action to take when 66 (or above) reaches beta.

I filed bug 1520261 to track the Releng work when ARM64 Fennec is ready to ride the trains.

Comment 28

[Intermittent Failures Robot]

2 failures in 2378 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• Android: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1368484&startday=2019-01-14&endday=2019-01-20&tree=all

Comment 29

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/b01dbd843889

Comment 30

[Chris Peterson [:cpeterson]]

(In reply to Tom Prince [:tomprince] from comment #29)

https://hg.mozilla.org/releases/mozilla-esr60/rev/b01dbd843889

Tom, why did these Android changes need to be uplifted to ESR 60? ARM64 Fennec Nightly was enabled in 66, not 60, and we don't publish any ESR Fennec builds to the Google Play Store anyway.

Comment 31

[Tom Prince [:tomprince]]

They probably didn't strictly need to be uplifted. Each release, I've been uplifting changes to taskgraph and release related code to esr60 (see here). This patch got uplifted, as it makes it easier to look at the diff between esr and m-c, and since, as you mentioned, we don't build fennec on ESR, it is particularly low risk.

Comment 32

[Chris Peterson [:cpeterson]]

(In reply to Tom Prince [:tomprince] from comment #31)

They probably didn't strictly need to be uplifted. Each release, I've been uplifting changes to taskgraph and release related code to esr60 (see here). This patch got uplifted, as it makes it easier to look at the diff between esr and m-c, and since, as you mentioned, we don't build fennec on ESR, it is particularly low risk.

Sounds good. I just wanted to make sure there wasn't some AArch64 Fennec problem that I hadn't heard about yet. :)