1117241 - add Aurora support for split-apk builds in advance of next release merge

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Jordan Lund (:jlund)]

I think all that is required is:

1) a gecko version bump in buildbot-configs
2) support for m-a single locale configs (l10n nightlies)
3) balrog rule support

Comment 2

[Richard Newman [:rnewman]]

Jordan, do you mean uplifting to Aurora 36, or just prepping for the next step in the current train cycle?

Comment 3

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #2)
> Jordan, do you mean uplifting to Aurora 36, or just prepping for the next
> step in the current train cycle?

apologies, I was going to uplift this but since next merge is in ~10 days, I think I'm going to let this ride trains naturally.

Comment 4

[Jordan Lund (:jlund)]

Attachment: 150102_bug_1117241_split_apk_aurora-mozharness.patch

same as the non split configs but pointing to split-apk for mozconfigs and branch name.

similar process to what I did for m-c

Comment 5

[Jordan Lund (:jlund)]

> 1) a gecko version bump in buildbot-configs

we are no longer uplifting, just letting trains ride so this step isn't needed

> 2) support for m-a single locale configs (l10n nightlies)

done here with comment 4

> 3) balrog rule support

this one is a bit tricky. we will have to lock fennec aurora rule on merge day, and then add a 'Fennec-mozilla-aurora-api-9-nightly-latest' rule with OS_VERSION:'2.3' and a higher priority then: 'Fennec-mozilla-aurora-nightly-latest'. this is the exact process as mozilla-central. the reason I can't add the rules now is because the split doesn't exist on aurora yet.

I will be on holidays for this merge. There is a chance I may work merge day to handle things like this but in case I don't, ben, do you think you can add the rule that day? I believe next release is on Jan 13th.


ni: ben wrt ^ balrog rule step

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Jordan Lund (:jlund) PTO till Jan 14th from comment #5)
> > 1) a gecko version bump in buildbot-configs
> 
> we are no longer uplifting, just letting trains ride so this step isn't
> needed
> 
> > 2) support for m-a single locale configs (l10n nightlies)
> 
> done here with comment 4
> 
> > 3) balrog rule support
> 
> this one is a bit tricky. we will have to lock fennec aurora rule on merge
> day, and then add a 'Fennec-mozilla-aurora-api-9-nightly-latest' rule with
> OS_VERSION:'2.3' and a higher priority then:
> 'Fennec-mozilla-aurora-nightly-latest'. this is the exact process as
> mozilla-central. the reason I can't add the rules now is because the split
> doesn't exist on aurora yet.
> 
> I will be on holidays for this merge. There is a chance I may work merge day
> to handle things like this but in case I don't, ben, do you think you can
> add the rule that day? I believe next release is on Jan 13th.
> 
> 
> ni: ben wrt ^ balrog rule step

That sounds right to me. Rail is handling the merge though - seem okay to you, Rail?

Comment 7

[Rail Aliiev [:rail]]

sure, I'll take care of this or ask for help.

Comment 8

[Kim Moir [:kmoir] ET]

Comment on attachment 8543457 [details] [diff] [review]
150102_bug_1117241_split_apk_aurora-mozharness.patch

lgtm, I noticed there's whitespace here

30 'balrog_credentials_file': 'oauth.txt',    <---

31 "tools_repo": "https://hg.mozilla.org/build/tools",

Comment 9

[Jordan Lund (:jlund)]

Comment on attachment 8543457 [details] [diff] [review]
150102_bug_1117241_split_apk_aurora-mozharness.patch

thanks kim! removed added whitespace.

I've landed this on default so it doesn't bust l10n nightly builds after the merge: https://hg.mozilla.org/build/mozharness/rev/5fe2aaab2c6a

I'll be flying on merge day.

Comment 10

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/5fe2aaab2c6a

Comment 11

[Mark Finkle (:mfinkle) (use needinfo?)]

This patch seems to be using android-api-9 and android-api-11 mozconfigs. Shouldn't we be using android-api-9-10 and android-api-11 mozconfigs?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

The single-locale l10n jobs are failing for API 9, wrong mozconfig definition I think:

https://tbpl.mozilla.org/php/getParsedLog.php?id=56819588&tree=Mozilla-Aurora&full=1
Android armv7 API 9 mozilla-aurora l10n nightly-1
02:11:31    ERROR - Can't copy /builds/slave/m-aurora-and-api-9-l10n_1-0000/build/mozilla-aurora/mobile/android/config/mozconfigs/android-api-9/l10n-nightly to /builds/slave/m-aurora-and-api-9-l10n_1-0000/build/mozilla-aurora/.mozconfig: [Errno 2] No such file or directory: '/builds/slave/m-aurora-and-api-9-l10n_1-0000/build/mozilla-aurora/mobile/android/config/mozconfigs/android-api-9/l10n-nightly'!

And for 10+ too looks, like it didn't get the mozconfig updated in http://hg.mozilla.org/releases/mozilla-aurora/rev/300ca104f8fb#l17.1:
https://tbpl.mozilla.org/php/getParsedLog.php?id=56819617&tree=Mozilla-Aurora&full=1
Android armv7 API 10+ mozilla-aurora l10n nightly-1
02:16:38     INFO -  checking Python environment is Mozilla virtualenv... yes
02:16:38     INFO -  configure: error: Invalid value --with-l10n-base, ../../l10n-central doesn't exist

Comment 13

[Rail Aliiev [:rail]]

Attachment: fix_android_l10n-configs-aurora.diff

This will fix uplift config changes (will be fixed as a part of bug 1120969).

[Feature/regressing bug #]: l10n fails to built
[User impact if declined]: no split apk l10n builds now :/
[Describe test coverage new/current, TBPL]: 
[Risks and why]: low
[String/UUID change made/needed]: none

Comment 14

[Jordan Lund (:jlund)]

(In reply to Mark Finkle (:mfinkle) from comment #11)
> This patch seems to be using android-api-9 and android-api-11 mozconfigs.
> Shouldn't we be using android-api-9-10 and android-api-11 mozconfigs?

yes, thanks for the catch

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8551362 [details] [diff] [review]
fix_android_l10n-configs-aurora.diff

Review of attachment 8551362 [details] [diff] [review]:
-----------------------------------------------------------------

more reason to get rid of the original and now not used mozconfigs: this patch changes android-api-10 but we should be fixing android-api-10/l10n-nightly

I'll handle https://bugzil.la/1121599 today

Comment 16

[Jordan Lund (:jlund)]

Attachment: fix for part 1 of comment 12

Approval Request Comment
[Feature/regressing bug #]:
[User impact if declined]:
[Describe test coverage new/current, TBPL]:
[Risks and why]: 
[String/UUID change made/needed]:

Comment 17

[Rail Aliiev [:rail]]

(In reply to Jordan Lund (:jlund) from comment #15)
> Comment on attachment 8551362 [details] [diff] [review]
> fix_android_l10n-configs-aurora.diff
> 
> Review of attachment 8551362 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> more reason to get rid of the original and now not used mozconfigs: this
> patch changes android-api-10 but we should be fixing
> android-api-10/l10n-nightly
> 
> I'll handle https://bugzil.la/1121599 today

It's great to get rid of unused stuff. I still think that the l10n path should be fixed in mobile/android/config/mozconfigs/android-api-9-10-constrained/l10n-nightly

Comment 18

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #15)
> Comment on attachment 8551362 [details] [diff] [review]
> fix_android_l10n-configs-aurora.diff
> 
> Review of attachment 8551362 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> more reason to get rid of the original and now not used mozconfigs: this
> patch changes android-api-10 but we should be fixing
> android-api-10/l10n-nightly
> 

bah, I mean "this patch changes android-api-10/l10n-nightly but we should be fixing android-api-11/l10n-nightly"

Comment 19

[Jordan Lund (:jlund)]

> It's great to get rid of unused stuff. I still think that the l10n path
> should be fixed in
> mobile/android/config/mozconfigs/android-api-9-10-constrained/l10n-nightly

yup, l10n path needs to be fixed in android-api-9-10-constrained/l10n-nightly and android-api-11/l10n-nightly

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8551422 [details] [diff] [review]
fix for part 1 of comment 12

+1 to removing unused configs, both in mh and gecko.

Don't need approval for this, so perhaps there's a gecko patch to follow that modifies 
 mobile/android/config/mozconfigs/android-api-11/l10n-nightly
 mobile/android/config/mozconfigs/android-api-9-10-constrained/l10n-nightly

Comment 21

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8551422 [details] [diff] [review]
fix for part 1 of comment 12

Aurora+

Comment 22

[Jordan Lund (:jlund)]

Attachment: fix l10n-base on aurora for current split mozconfigs

same as yours rails but ignores the soon to be deleted old mozconfigs and adds android-api-11/l10n-nightly

Comment 23

[Jordan Lund (:jlund)]

Comment on attachment 8552191 [details] [diff] [review]
fix l10n-base on aurora for current split mozconfigs

Approval Request Comment
[Feature/regressing bug #]: no bug but this
[User impact if declined]: no split apk l10n builds now
[Describe test coverage new/current, TreeHerder]:
[Risks and why]: low
[String/UUID change made/needed]: none

sorry lmandel or whovever, should have requested this patch for approval not the mozhaarness one approved in comment 21. I thought I needed this flag for anything affecting aurora in or out of tree.

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8552191 [details] [diff] [review]
fix l10n-base on aurora for current split mozconfigs

Review of attachment 8552191 [details] [diff] [review]:
-----------------------------------------------------------------

The patch won't apply unless you remove \r characters (windows?) sed -i 's/\r//' the_patch.diff
Otherwise it looks good to me.

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8551422 [details] [diff] [review]
fix for part 1 of comment 12

default:   https://hg.mozilla.org/build/mozharness/rev/89f0bc25dcfe

Comment 26

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/89f0bc25dcfe

Comment 27

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8552191 [details] [diff] [review]
fix l10n-base on aurora for current split mozconfigs

Ok. Keep pushing to get those split APKs going. Aurora+

Comment 28

[Jordan Lund (:jlund)]

Comment on attachment 8552191 [details] [diff] [review]
fix l10n-base on aurora for current split mozconfigs

thanks!

https://hg.mozilla.org/releases/mozilla-aurora/rev/5b21e1bf0ee7

Comment 29

[Jordan Lund (:jlund)]

I'll resolve this bug once l10n builds look healthy again

Comment 30

[Jordan Lund (:jlund)]

l10n m-a infra looks good for both apk builds again however chunk 3 is consistently failing for both. I've reopened 1057025 to track chunk 3 failure but will close this as I think it is unrelated