Closed Bug 1121160 Opened 10 years ago Closed 10 years ago

Release automation support for split-apk builds

Categories

(Release Engineering :: Release Automation: Other, defect)

Product:
Release Engineering
Component:
Release Automation: Other
Platform:
x86
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox37 fixed, firefox38 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox37 --- fixed
firefox38 --- fixed

People

(Reporter: nthomas, Assigned: bhearsum)

References

Details

Attachments

(5 files, 1 obsolete file)

split_apk_merge_day_scripts.diff
1.09 KB, patch
Details | Diff | Splinter Review
150129_beta_splitapk-mh.patch
21.27 KB, patch
bhearsum
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
update beta buildbot configs for split apk work
9.71 KB, patch
jlund
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
whitelist updates for api 9/11
3.05 KB, patch
jlund
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
add release mozconfigs for split apk bulids
3.41 KB, patch
mfinkle
: review+
rnewman
: review+
lmandel
: approval-mozilla-aurora+
bhearsum
: checked-in+
Details | Diff | Splinter Review
We need to enable these platforms in the release configs, test in staging etc, split-apk builds have just ridden the train to Aurora, and assuming they ride again we'll need this for the Feb 23 merge.
Depends on: 1120969
Whenever we have "release" and "l10n-release" mozconfigs in place we can land this patch to enable automatic beta-to-release mozconfig magic.
here is what i have for the single locale beta support. this should take care of staging and prod release. note: multi locale config support does not need mh change for beta releases here is an interdiff of what is changing in the respective new mh configs in contrast to the nonsplit equivalent: http://people.mozilla.org/~jlund/beta_splitapk-interdiff.diff the keys I am unsure about are whether we need to change: - 'locales_platform' - 'base_post_upload_cmd'
Attached patch 150129_beta_splitapk-m-b.patch (obsolete) — — Splinter Review
here is the m-b mozconfigs needed in accordance to http://pastebin.mozilla.org/8424374
(In reply to Jordan Lund (:jlund) from comment #3) > Created attachment 8556650 [details] [diff] [review] > 150129_beta_splitapk-m-b.patch > > here is the m-b mozconfigs needed in accordance to > http://pastebin.mozilla.org/8424374 Looks like we need a bunch of whitelist updates both to the in-repo and build/tools whitelist: 2015-01-30 08:52:01,153 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/release: ac_add_options --enable-profiling 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/release: export MOZ_DISABLE_GECKOVIEW=1 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/release but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly: ac_add_options --with-branding=mobile/android/branding/beta 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/release: ac_add_options --with-branding=mobile/android/branding/nightly 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/release but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly: ac_add_options --enable-updater 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/release: STRIP_FLAGS="--strip-debug" 2015-01-30 08:52:01,154 : ERROR : found in mobile/android/config/mozconfigs/android-api-9-10-constrained/release but not in mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly: ac_add_options --enable-official-branding 2015-01-30 08:52:02,516 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/nightly but not in mobile/android/config/mozconfigs/android-api-11/release: ac_add_options --enable-profiling 2015-01-30 08:52:02,517 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/release but not in mobile/android/config/mozconfigs/android-api-11/nightly: ac_add_options --with-branding=mobile/android/branding/beta 2015-01-30 08:52:02,517 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/nightly but not in mobile/android/config/mozconfigs/android-api-11/release: ac_add_options --with-branding=mobile/android/branding/nightly 2015-01-30 08:52:02,517 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/release but not in mobile/android/config/mozconfigs/android-api-11/nightly: ac_add_options --enable-updater 2015-01-30 08:52:02,517 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/nightly but not in mobile/android/config/mozconfigs/android-api-11/release: STRIP_FLAGS="--strip-debug" 2015-01-30 08:52:02,517 : ERROR : found in mobile/android/config/mozconfigs/android-api-11/release but not in mobile/android/config/mozconfigs/android-api-11/nightly: ac_add_options --enable-official-branding 2015-01-30 08:52:03,937 : ERROR : Error verifying mozconfigs I'll try to sort these out and attach a new patch.
Some fixes were needed already, but there's still a problem where the repacks aren't submitting to balrog because they're using the wrong buildbot configs. dunno why that is, still need to look into it.
This is what we needed to get the staging release to work.
Attachment #8559963 - Flags: review?(jlund)
Comment on attachment 8556647 [details] [diff] [review] 150129_beta_splitapk-mh.patch This patch worked perfect in staging.
Attachment #8556647 - Flags: review+
I got confirmation from rnewman and nalexander that we *do* want to disable geckoview for all l10n builds. so, iow, we want: `export MOZ_DISABLE_GECKOVIEW=1` in all l10n mozconfigs. This applies for both api-9-10 and api-11. snippet: 14:31:21 <jlund|buildduty> rnewman: but in the l10n mozconfig we had it http://hg.mozilla.org/mozilla-central/file/0311e3b1c374/mobile/android/config/mozconfigs/android/l10n-nightly 14:31:47 <•rnewman> oh, different question 14:32:13 <•rnewman> jlund|buildduty: I don't think we care about making GeckoView builds in l10n nightly 14:32:21 <•rnewman> and we can't for constrained builds 14:32:34 <•rnewman> so both of those two axes should have M_D_G 14:32:42 <•mfinkle> right, we don't care about geckoview builds in l10n nightlies 14:33:15 <jlund|buildduty> rnewman: great we will leave it in for both 9-10 and 11 l10n. ty for confirmation
Comment on attachment 8559963 [details] [diff] [review] update beta buildbot configs for split apk work Review of attachment 8559963 [details] [diff] [review]: -----------------------------------------------------------------
Attachment #8559963 - Flags: review?(jlund) → review+
It's a little unclear why the branding ones are needed...I want to look into that a bit more.
Attachment #8560532 - Flags: review?(jlund)
These are mostly the same as the nightly equivalents. There's still some confusion about MOZ_DISABLE_GECKOVIEW, though. I don't quite understand why it's in the nightly api 11 mozconfig (or if it even does anything there). It's clearly required for the api 9 mozconfigs, though.
Attachment #8560578 - Flags: review?(mark.finkle)
Comment on attachment 8556647 [details] [diff] [review] 150129_beta_splitapk-mh.patch This patch worked 100% fine in staging. I've landed it. We'll still need one for mozilla-release at some point.
Attachment #8556647 - Flags: checked-in+
Attachment #8556650 - Attachment is obsolete: true
Comment on attachment 8560578 [details] [diff] [review] add release mozconfigs for split apk bulids MOZ_DISABLE_GECKOVIEW=1 is only in the l10n "android-api-11" mozconfig. It's not in the non-l10n mozconfig. Adding Richard for a second look.
Attachment #8560578 - Flags: review?(rnewman)
Attachment #8560578 - Flags: review?(mark.finkle)
Attachment #8560578 - Flags: review+
Comment on attachment 8560532 [details] [diff] [review] whitelist updates for api 9/11 Review of attachment 8560532 [details] [diff] [review]: ----------------------------------------------------------------- I'm probably not the best to stamp this. I'm a little confused about what the 'whitelist' purpose is. I can verify that we are adding the necessary platforms but I'm not sure which mozconfig options those platforms should be copying from. ::: buildbot-helpers/mozconfig_whitelist @@ +88,5 @@ > + 'ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}', > + 'ac_add_options --enable-warnings-as-errors', > + 'ac_add_options --with-ccache=/usr/bin/ccache', > + 'ac_add_options --enable-profiling', > + 'ac_add_options --disable-elf-hack # --enable-elf-hack conflicts with --enable-profiling', Taking a look at mozconfigs, I can't see where we have --disable-elf-hack anywhere. @@ +162,5 @@ > "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'", > ] > whitelist[branch]['android'] += ['ac_add_options --enable-updater'] > + whitelist[branch]['android-api-9'] += ['ac_add_options --enable-updater'] > + whitelist[branch]['android-api-11'] += ['ac_add_options --enable-updater'] is this future proofing m-b -> m-r ?
Comment on attachment 8560578 [details] [diff] [review] add release mozconfigs for split apk bulids Review of attachment 8560578 [details] [diff] [review]: ----------------------------------------------------------------- This looks fine to me. ::: mobile/android/config/mozconfigs/android-api-11/release @@ +3,5 @@ > +# Android > +ac_add_options --with-android-min-sdk=11 > +ac_add_options --target=arm-linux-androideabi > + > +ac_add_options --with-branding=mobile/android/branding/beta You'll know better than me, but should this be 'beta'?
Attachment #8560578 - Flags: review?(rnewman) → review+
Comment on attachment 8560532 [details] [diff] [review] whitelist updates for api 9/11 Review of attachment 8560532 [details] [diff] [review]: ----------------------------------------------------------------- r- just to clear my queue. I'll switch to r+ once I touch base with Ben and have the patch explained.
Attachment #8560532 - Flags: review?(jlund) → review-
(In reply to Richard Newman [:rnewman] from comment #15) > Comment on attachment 8560578 [details] [diff] [review] > add release mozconfigs for split apk bulids > > Review of attachment 8560578 [details] [diff] [review]: > ----------------------------------------------------------------- > > This looks fine to me. > > ::: mobile/android/config/mozconfigs/android-api-11/release > @@ +3,5 @@ > > +# Android > > +ac_add_options --with-android-min-sdk=11 > > +ac_add_options --target=arm-linux-androideabi > > + > > +ac_add_options --with-branding=mobile/android/branding/beta > > You'll know better than me, but should this be 'beta'? Yeah. The "release" name is a touch confusing, it means "release style builds" which Betas are. When we uplift to mozilla-release there's scripts that adjust this.
Comment on attachment 8560578 [details] [diff] [review] add release mozconfigs for split apk bulids Drivers, this is a no-op change that's needed to support split apk builds on mozilla-beta, so we need to get it on mozilla-aurora before the uplift.
Attachment #8560578 - Flags: approval-mozilla-aurora?
(In reply to Jordan Lund (:jlund) from comment #14) > Comment on attachment 8560532 [details] [diff] [review] > whitelist updates for api 9/11 > > Review of attachment 8560532 [details] [diff] [review]: > ----------------------------------------------------------------- > > I'm probably not the best to stamp this. I'm a little confused about what > the 'whitelist' purpose is. I can verify that we are adding the necessary > platforms but I'm not sure which mozconfig options those platforms should be > copying from. The whitelist is used as part of release_sanity.py. Part of that script makes sure that the nightly and release mozconfigs are exactly the same, except for allowed differences (the ones listed in the whitelist). This protects against building incorrectly on beta or release. In the past we've forgotten to include necessary changes (eg, it would've been easy to miss the MOZ_DISABLE_GECKOVIEW stuff without release sanity). If you're not comfortable reviewing this I can get somebody else to do it, no worries :). > ::: buildbot-helpers/mozconfig_whitelist > @@ +88,5 @@ > > + 'ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}', > > + 'ac_add_options --enable-warnings-as-errors', > > + 'ac_add_options --with-ccache=/usr/bin/ccache', > > + 'ac_add_options --enable-profiling', > > + 'ac_add_options --disable-elf-hack # --enable-elf-hack conflicts with --enable-profiling', > > Taking a look at mozconfigs, I can't see where we have --disable-elf-hack > anywhere. This is still something that differs between nightly and release+esr (http://mxr.mozilla.org/mozilla-esr31/search?string=--disable-elf-hack, http://mxr.mozilla.org/mozilla-release/search?string=--disable-elf-hack). > @@ +162,5 @@ > > "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'", > > ] > > whitelist[branch]['android'] += ['ac_add_options --enable-updater'] > > + whitelist[branch]['android-api-9'] += ['ac_add_options --enable-updater'] > > + whitelist[branch]['android-api-11'] += ['ac_add_options --enable-updater'] > > is this future proofing m-b -> m-r ? Yep. May as well do it now, saves trouble later.
Flags: needinfo?(jlund)
Comment on attachment 8560578 [details] [diff] [review] add release mozconfigs for split apk bulids Aurora+
Attachment #8560578 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/8090a0b68400
Assignee: nobody → bhearsum
Status: NEW → RESOLVED
Closed: 10 years ago
status-firefox38: --- → fixed
Resolution: --- → FIXED
Attachment #8560532 - Flags: review- → review+
> If you're not > comfortable reviewing this I can get somebody else to do it, no worries :). makes sense r+ > > > ::: buildbot-helpers/mozconfig_whitelist > > @@ +88,5 @@ > > > + 'ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}', > > > + 'ac_add_options --enable-warnings-as-errors', > > > + 'ac_add_options --with-ccache=/usr/bin/ccache', > > > + 'ac_add_options --enable-profiling', > > > + 'ac_add_options --disable-elf-hack # --enable-elf-hack conflicts with --enable-profiling', > > > > Taking a look at mozconfigs, I can't see where we have --disable-elf-hack > > anywhere. > > This is still something that differs between nightly and release+esr > (http://mxr.mozilla.org/mozilla-esr31/search?string=--disable-elf-hack, > http://mxr.mozilla.org/mozilla-release/search?string=--disable-elf-hack). oh, sorry, I thought: whitelist['nightly'] meant that this was whitelisting for only m-c and related branches
Flags: needinfo?(jlund)
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Attachment #8560532 - Flags: checked-in+
Attachment #8559963 - Flags: checked-in+
We should be all set here now for 37.0b1. The mozilla-release release configs will need updating in ~5 weeks, but I'll take care of that in bug 1119237.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Depends on: 1151498
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: