1122059 - Use different filenames for each variant Android APK

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Sylvestre Ledru [:Sylvestre]]

Files have the same name (fennec-38.0a1.multi.android-arm.apk):
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-9/

This is a problem for release management:
* our tools does not support that (ok, that is not blocking but it would be harder to manage files with the same names)
* that introduces risks that we upload twice the same file

On the more general perspective, it introduces some potential confusions.

Comment 1

[Nick Alexander :nalexander [he/him]]

Can we use this bug to clean up releng's APK handling?  The scripts do all kinds of special handling to find gecko-*.apk and robocop.apk.  I want to upload lots of APK names and do different things with them in automation; for example, I want to upload both Proguarded and non-Proguarded APKs and use the non-Proguarded APKs in certain tests (JUnit 3 tests).  I've run into tickets like Bug 919627 trying such things.  And don't get me started on the nightmare of code that deploys robocop.apk to testing devices!

I propose that we include some {key: "file.apk"} in the tree configurations, either per mozharness job or in some other central location.  I do not know if this is hard to achieve; I think for mozharness, it is not, but mozharness is only some small part of the issue.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=919627

Comment 2

[Sylvestre Ledru [:Sylvestre]]

(In reply to Nick Alexander :nalexander from comment #1)
> Can we use this bug to clean up releng's APK handling?  
Sure, I was just explaining our (relman) expectations. Seems there are more ;)

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Chris, can you find someone to help with this bug?
We need a fix for this for the 37 cycle.

Comment 4

[Chris Cooper [:coop] (he/him)]

Assigning to :jlund to wrap-up the splitapk work.

Comment 5

[Jordan Lund (:jlund)]

as mentioned here: https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3 releng doesn't determine the apk name, we simply look for an apk file that was generated: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#203

aiui, mobile team was experimenting with different package names last week here: "Bug 1120762 - Test, solidify, and document Google Play distribution approach for split APKs"

I'm not sure if they changed any filenames but either way, I'll keep this bug moving by chatting to names mentioned here: http://hg.mozilla.org/mozilla-central/annotate/eea6188b9b05/mobile/android/build.mk#l37

Comment 6

[Sylvestre Ledru [:Sylvestre]]

Richard, can you help with the new apk file name? Thanks
We need that to update our scripts.

Comment 7

[Richard Newman [:rnewman]]

Package names != filenames. I haven't touched the filenames, nor have I any idea how that works. Nick might.

Comment 8

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #7)
> Package names != filenames. I haven't touched the filenames, nor have I any
> idea how that works. Nick might.

sorry I meant you were changing package names and I wasn't sure if you experimented with changing filenames too.

I don't want to just try and hand responsibility off. That doesn't make sense since I started the apk split mess :)

Maybe if someone like nick (I also see mbrubeck, mfinkle, and blassey on build.mk contributor list) can get me up to speed with how we set filenames or enlighten me on any consequences of changing lines like http://hg.mozilla.org/mozilla-central/annotate/eea6188b9b05/mobile/android/build.mk#l37 from the in-tree compile side of things, that would be appreciated :D

nick, can you help me or point me to someone who may know more? thanks :)

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Jordan Lund (:jlund) from comment #8)
> (In reply to Richard Newman [:rnewman] from comment #7)
> > Package names != filenames. I haven't touched the filenames, nor have I any
> > idea how that works. Nick might.
> 
> sorry I meant you were changing package names and I wasn't sure if you
> experimented with changing filenames too.

We have not.  I have tried uploading multiple APK files and been sorely disappointed: Bug 919627, although that code looked to have changed when I last looked.

I have always been too scared to change APK file names since so much depends on "magic pattern matching": the build system has expectations, but the interaction between the build system and releng, and the updaters, also does.  And things that scrape ftp.m.o.  There's a long tail to chase down.

> I don't want to just try and hand responsibility off. That doesn't make
> sense since I started the apk split mess :)
> 
> Maybe if someone like nick (I also see mbrubeck, mfinkle, and blassey on
> build.mk contributor list) can get me up to speed with how we set filenames
> or enlighten me on any consequences of changing lines like
> http://hg.mozilla.org/mozilla-central/annotate/eea6188b9b05/mobile/android/
> build.mk#l37 from the in-tree compile side of things, that would be
> appreciated :D
> 
> nick, can you help me or point me to someone who may know more? thanks :)

All of the filename stuff is controlled by https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk.  Package name here is historical and refers to the name of the build system output binary; it does /not/ refer to the "Android package name", which is a Java-style org.mozilla.* reverse-domain identifier.

I would talk to glandium.

Comment 10

[Jordan Lund (:jlund)]

pinged glandium over irc. adding ni here

Comment 11

[Mike Hommey [:glandium]]

The build system actually has no expectations about the apk file name. Everything that does is on the releng side.

Comment 12

[Jordan Lund (:jlund)]

Attachment: 150304_bug_1122059_use_diff_apk_name_for_split_m-c-2.patch

I'm giving this a whirl here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff9094b80b76

my assumption is that if a min sdk version is set, then it is a split build: either api-9-10 or api-11-* and we should append the apk name with 'api-MIN_SDK'

I'm hoping for the x86 android case, this will be ignored.

Although, this way seems fragile and bear in mind, this could be my first make patch ;)

Comment 13

[Justin Wood (:Callek)]

Jordan,

I'm reluctantly showing I'm familiar with the build system here, I was going to recommend MOZ_PKG_SPECIAL in the relevant mozconfigs (like asan)

*however* it looks like android uses that as something else, specifically changing stuff for updater, as well as marking it as a low memory device.

(b2g uses it for other reasons too) http://mxr.mozilla.org/build/search?string=MOZ_PKG_SPECIAL&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=build

So your solution might indeed be the best bandaid afterall....

Comment 14

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #12)
> Created attachment 8573066 [details] [diff] [review]
> 150304_bug_1122059_use_diff_apk_name_for_split_m-c-2.patch
> 
> I'm giving this a whirl here:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff9094b80b76
> 
ok, so that seemed to work as expected: apk generated fennec-39.0a1.en-US.android-arm-api-11.apk

but this breaks automation. looks like I need to change our regex here: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#629

it looks for *arm.apk and we are appending to arm: arm-api-{9,11}.apk

I should be able to navigate to the appropriate fix from here :)

Comment 15

[Jordan Lund (:jlund)]

was pulled away for higher priorities. I had some time on Monday and today to look at this again. I'm currently trying to test this in staging.

Unfortunately, android builds still rely on buildbot factories (MozillaBuildFactory) and not mozharness so testing is harder.

At any rate, I'll report back on this tomorrow.

Comment 16

[Jordan Lund (:jlund)]

I've got a build working in staging but it is unconfirmed with testing. The issue here is controlling this rollout in a safe manner. I think we can do one of two things:

1) configure buildbot's android build factory to change the filename on one project branch and update mozharness tests to accept to accept both the old and new filenames everywhere. Then configure the new filename across trunk.

2) wait till end of quarter and address this when android builds are in mozharness + taskcluster: https://bugzil.la/1118394

the unknowns here is how this will affect our nightly, post upload automation, and release automation. e.g. Will we need to update bouncer links for grabbing nightlies? This is the area that I'm unfamiliar with and will be a pain dealing with while android builds are driven from buildbot. Unless this is blocking a release, my vote is to wait for mozharness+taskcluster builds and deal with this along side: https://bugzil.la/1122509

 Sylvester: does that make sense? Rather than putting effort in tmp fixing this here, shall we wait for the multiple taskcluster efforts in Q2?

Comment 17

[Jordan Lund (:jlund)]

fyi - I will be on PTO until May 13th.

the patch(es) I was playing with was: https://bugzilla.mozilla.org/attachment.cgi?id=8573066

and from buildbot-custom:

diff --git a/process/factory.py b/process/factory.py
index 0063b09..916cc9f 100644
--- a/process/factory.py
+++ b/process/factory.py
@@ -626,6 +626,12 @@ class MozillaBuildFactory(RequestSortingBuildFactory, MockMixin):
             packageFilename = '*arm-armv6.apk'
         elif 'android-x86' in self.complete_platform:
             packageFilename = '*android-i386.apk'
+        elif 'android-api-9' in self.complete_platform:
+            # the arm.apk is to avoid unsigned/unaligned apks
+            packageFilename = '*arm-api-9.apk'
+        elif 'android-api-11' in self.complete_platform:
+            # the arm.apk is to avoid unsigned/unaligned apks
+            packageFilename = '*arm-api-11.apk'
         elif 'android' in self.complete_platform:
             # the arm.apk is to avoid unsigned/unaligned apks
             packageFilename = '*arm.apk'

Comment 18

[Sylvestre Ledru [:Sylvestre]]

(In reply to Jordan Lund (:jlund) from comment #16)
>  Sylvester: does that make sense? Rather than putting effort in tmp fixing
> this here, shall we wait for the multiple taskcluster efforts in Q2?
There is no rush here. If we are going to setup a clean and scalable solution, I would rather wait for it!

Comment 19

[Sylvestre Ledru [:Sylvestre]]

Jordan, any update on this? We are discussing with a partner and this would help to avoid confusion.

Comment 20

[Jordan Lund (:jlund)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #19)
> Jordan, any update on this? We are discussing with a partner and this would
> help to avoid confusion.

android builds use taskcluster + mozharness now for m-c and m-a nightlies. We could probably change or at least experiment with changing the apk name pretty easily with those but it hasn't landed on m-b/m-r yet. Those repos are being blocked by and waiting on https://bugzil.la/1118794

iow we could proceed with switching m-c/m-a but that will result in a divergence in apk names in release branches for a while which I would assume would be confusing for release configs? Please needinfo me how you would like to proceed

Comment 21

[Sylvestre Ledru [:Sylvestre]]

Now that we have much more automation here, this is no longer needed.