Closed Bug 1080749 Opened 10 years ago Closed 9 years ago

Add nightly jobs for new splitapk Android builders

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(fennec37+)

RESOLVED FIXED
Tracking Status
fennec 37+ ---

People

(Reporter: jlund, Assigned: jlund)

References

Details

Attachments

(16 files, 3 obsolete files)

3.00 KB, patch
hwine
: review+
Details | Diff | Splinter Review
4.58 KB, patch
hwine
: review+
Details | Diff | Splinter Review
614 bytes, patch
nthomas
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
2.39 KB, patch
bhearsum
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
1.16 KB, patch
kmoir
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
4.60 KB, patch
kmoir
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
2.02 KB, patch
kmoir
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
6.89 KB, patch
jlund
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
5.29 KB, patch
coop
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
10.53 KB, patch
kmoir
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
2.09 KB, patch
rnewman
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
5.07 KB, patch
coop
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
867 bytes, patch
kmoir
: review+
Details | Diff | Splinter Review
701 bytes, patch
nthomas
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
623 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
11.25 KB, patch
kmoir
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
we are currently splitting our armv7 android builds into two based on API levels (releng bug 1073772).

This means that we will have two new nightlies that need to run and support updates with a balrog.

This bug tracks that effort.

current issues:
    1) we need a new build target. 'android'[1] is currently 'Android_arm-eabi-gcc3'[2] but we need to support two new ones and figure out how to differienciate which is which for our update server. 'android' is going to ride the trains so we will want to keep support for that and left untouched. My understanding is that we can pivot off certain values (version, platform, etc) that are defined from in tree at compile time. These values can be used to identify what target we are building. This will have to be coordinated between devs (adding the identifiers) and releng (supporting the target in our update server, Balrog).

rnewman: is any of this familiar to you or do you know who I can coordinate with? Maybe someone who helped define 'Android_arm-eabi-gcc3'?

    2) in line with this, I am also unsure how we are going to switch current users over to the new nightly builds once this hits m-c. e.g. how do we serve the new nightly updates to users who were on 'android' but now need the appropriate new one. I know it will be based on api version, but how will balrog know what to serve? How do we change the nightly channel and will the new channel apply? Lot's of questions here.

bhearsum, do you have any insight to this?

    3) I have one other current issue that's failing but I think I can solve that on my own. mentioning it just to document: there is a step in nightlies where we run a mozharness script: 'mozharness/scripts/multil10n.py'. that is failing because it looks for a json file that is derived from the builder name. I'll look into creating that file.

error line: IOError: Can't find multi_locale/mozilla-central_android-api-10.json in ['.', '/builds/slave/m-cen-and-api-10-ntly-00000000/mozharness/scripts/../configs', '/builds/slave/m-cen-and-api-10-ntly-00000000/mozharness/scripts/../../configs']!


rollout wise, we should first enable these builds to a branch that supports nightlies and verify updates work before replacing m-c. since cedar is the place to iterate on new jobs (its where we are landing the opt/debug jobs of this new android split), we can also give cedar nightly abilities. ben has already confirmed that would be ok. I'll probably file a new bug for that.


[1] http://hg.mozilla.org/build/buildbot-configs/file/da9a1022d760/mozilla/config.py#l1335
[2] http://hg.mozilla.org/build/tools/file/2469042323a6/lib/python/release/platforms.py#l24
needinfo'ing for ^ comment 0
Flags: needinfo?(rnewman)
Flags: needinfo?(bhearsum)
(In reply to Jordan Lund (:jlund) from comment #0)
> we are currently splitting our armv7 android builds into two based on API
> levels (releng bug 1073772).
> 
> This means that we will have two new nightlies that need to run and support
> updates with a balrog.
> 
> This bug tracks that effort.
> 
> current issues:
>     1) we need a new build target. 'android'[1] is currently
> 'Android_arm-eabi-gcc3'[2] but we need to support two new ones and figure
> out how to differienciate which is which for our update server. 'android' is
> going to ride the trains so we will want to keep support for that and left
> untouched. My understanding is that we can pivot off certain values
> (version, platform, etc) that are defined from in tree at compile time.
> These values can be used to identify what target we are building. This will
> have to be coordinated between devs (adding the identifiers) and releng
> (supporting the target in our update server, Balrog).
> 
> rnewman: is any of this familiar to you or do you know who I can coordinate
> with? Maybe someone who helped define 'Android_arm-eabi-gcc3'?

It would help to understand what the real differences are. Eg, are we enabling or disabling features based on a minimum Android version only? Or are we doing it based on things that could be different per-phone, even if they run the same Android version.

Build target is certainly the recommended way to do this right now, but depending on what exactly we're talking about there may be other options.

It would also help to see what the update URLs look like for these already, if we have some builds that have been done by hand. This can be found by starting adb logcat and checking for updates (look for "aus4.mozilla.org").

>     2) in line with this, I am also unsure how we are going to switch
> current users over to the new nightly builds once this hits m-c. e.g. how do
> we serve the new nightly updates to users who were on 'android' but now need
> the appropriate new one. I know it will be based on api version, but how
> will balrog know what to serve? How do we change the nightly channel and
> will the new channel apply? Lot's of questions here.

I think the least risky thing to do here is to continue using the existing build target for the build with lowest requirements. This would mean a reinstall for users that are on higher end devices. Perhaps we could pop a whatsnew page or something for a period of time to point people at it.

We only have about ~10,000 people on Nightly+Aurora, so I don't think it's worth jumping through huge hoops here.
Flags: needinfo?(bhearsum)
(In reply to Jordan Lund (:jlund) from comment #0)

> rnewman: is any of this familiar to you or do you know who I can coordinate
> with? Maybe someone who helped define 'Android_arm-eabi-gcc3'?

I can copy and modify the 'android' build config in m-c.

I don't know anything about the hooks outside the tree, I'm afraid.
 

(In reply to Ben Hearsum [:bhearsum] from comment #2)

> It would help to understand what the real differences are. Eg, are we
> enabling or disabling features based on a minimum Android version only? Or
> are we doing it based on things that could be different per-phone, even if
> they run the same Android version.

The only discriminating factor right now is the current OS level of the device. If you're on API 9, the updater should give you the API 9 constrained build. If you're not, it should give you the API 10+ build.


Ideally if you upgrade Android on your device, the next update check will give you the correct APK. It's not clear whether our own updater can do this, because of course the updater in the old APK is now manifest-specified to not run on the upgraded OS. Google Play manages this out of band.

This is hard to test.


> We only have about ~10,000 people on Nightly+Aurora, so I don't think it's
> worth jumping through huge hoops here.

OTOH, they are high-value users.

We send Build.VERSION.RELEASE, which is good enough -- it'll be something like "2.3.3" -- so it seems like this is a problem we can solve without pushing work onto the user.
Flags: needinfo?(rnewman)
(In reply to Ben Hearsum [:bhearsum] from comment #2)

> It would help to understand what the real differences are. Eg, are we
> enabling or disabling features based on a minimum Android version only?

To go into a little detail here:

We're producing APKs with an AndroidManifest that specifies min and max SDK versions.

By doing so we implicitly guard portions of the code, so we can eliminate runtime checks and huge swathes of code that doesn't apply to the supported version.

For example, the API 9 APK won't include any code or resources to run on tablets, anything that uses Android Beam, etc. The API 10+ APK won't include Gingerbread-compat code. And so on.

The Gingerbread APK simply won't install on a modern phone. The modern APK won't install on a Gingerbread phone.

The Gingerbread APK also excludes some stuff (e.g., fonts) in order to save space. That's why we call it "API 9 constrained build".
(In reply to Richard Newman [:rnewman] from comment #3)
> > We only have about ~10,000 people on Nightly+Aurora, so I don't think it's
> > worth jumping through huge hoops here.
> 
> OTOH, they are high-value users.
> 
> We send Build.VERSION.RELEASE, which is good enough -- it'll be something
> like "2.3.3" -- so it seems like this is a problem we can solve without
> pushing work onto the user.

Can you give us a couple of sample update URLs where this is sent?
Here's the code:

        UPDATE_URL = "https://aus4.mozilla.org/update/4/" + AppConstants.MOZ_APP_BASENAME + "/" +
                     AppConstants.MOZ_APP_VERSION         +
                     "/%BUILDID%/Android_"                + AppConstants.MOZ_APP_ABI + pkgSpecial +
                     "/%LOCALE%/"                         + AppConstants.MOZ_UPDATE_CHANNEL +
                     "/%OS_VERSION%/default/default/"     + AppConstants.MOZILLA_VERSION +
                     "/update.xml";

%OS_VERSION% is substituted here:

        String url = UPDATE_URL.replace("%LOCALE%", locale).
            replace("%OS_VERSION%", Build.VERSION.RELEASE).
            replace("%BUILDID%", force ? "0" : AppConstants.MOZ_APP_BUILDID);

Build.VERSION.RELEASE is the human-readable Android OS version.

So here's what my phone requests:

https://aus4.mozilla.org/update/4/Fennec/35.0a1/0/Android_arm-eabi-gcc3/en-US/default/4.4.3/default/default/35.0a1/update.xml

That "4.4.3" is the bit we'd key off.
(In reply to Richard Newman [:rnewman] from comment #6)
> So here's what my phone requests:
> 
> https://aus4.mozilla.org/update/4/Fennec/35.0a1/0/Android_arm-eabi-gcc3/en-
> US/default/4.4.3/default/default/35.0a1/update.xml
> 
> That "4.4.3" is the bit we'd key off.

Okay. So it's possible to do this, though it's easier if that's in build target (the "Android_arm-eabi-gcc3" part). Jordan, we did something similar for Flame builds in https://bugzilla.mozilla.org/show_bug.cgi?id=1055305. That bug is a bit muddled now though, let me know if you want a hand decoding anything.
API 9 will be one of these strings:

"2.3"
"2.3.1"
"2.3.2"

We should probably consider adding the Android SDK_INT to the updater URL (in a new version) to avoid the need for this pain.
tracking-fennec: --- → 36+
tracking-fennec: 36+ → 35+
status update: I had a look at this again today and didn't get very far. I'm spending too much time on with unfamiliar parts. This is higher priority now so I am going to rope in some people knowledgeable with our update server for a braindump session tomorrow. I'll also be putting aside everything else until this is completed.
Blocks: 1100361
It's much appreciated, jlund.
Status: NEW → ASSIGNED
Summary: add nightly jobs for new splitapk android builders → Add nightly jobs for new splitapk Android builders
OK I've finished reading through the docs and source of balrog. This might be pretty trivial now that I'm familiar with the process.

Basically, IIUC, we need to do the following:

* enable the new split builds on a testing branch that does nightlies (or enable nightlies on cedar)
    ** I am going to add these to ash since 1) ash has nightlies already and I can play with the old non-split builds against split builds 2) adding split builds to new branches is easy
* add an ash rule that mirrors the m-c fennec non-split rule (this will serve as api-10+)
* add a new rule for the api-9 case (see below for specifics)
* once nightlies are verified (we can apply update to both 9 and 10+ builds) we can make the switch on m-c to use the new split and add the new new api-9 rule like we did for ash
* to be extra safe we should disable updates being served on the nightly m-c channel and use the nightlytest channel to verify new nightlies work as expected. we can lock nightly channel to previous nightly by: changing mapping to Fennec-mozilla-central-nightly-{prev_buildid} (same with new api-9 rule)
* verify new nightlies work, re-open fennec nightly channel: change mapping back to Fennec-mozilla-central-nightly-latest 

more specifically with the new api-9 rule:
comment 6 shows how we can pivot off OS_VERSION. Ideally we would like to change the URL that android requests so the build_target actually includes an API level. Currently build_target is Android_arm-eabi-gcc3, and is derived from {Android_" + AppConstants.MOZ_APP_ABI + pkgSpecial}). This allows us to not have to add a new mapping and submitter hack. However, as ben said in comment 7, we can do some magic on our end and I think this will be fastest solution. 

So basically we need to add:
mapping:Fennec-mozilla-central-api-9-nightly-latest | priority:92 | os_version:2.3,2.3.1,2.3.2
   * https://github.com/mozilla/balrog/blob/master/auslib/db.py#L692 suggests we can use multiple values split by comma for os_version

since we only know the exact OS_VERSION values of api-9 (2.3) are, technically api-9 will match both this new rule row and the old one so we need to give the api-9 one higher priority so it gets chosen.
   * see priority behaviour mentioned here: https://wiki.mozilla.org/Balrog/Rules

finally, we will have manipulate the submitter so we actually push bits to the new mappings. This will require a similar patch to like flame: http://hg.mozilla.org/build/tools/rev/9832742ed555

I'm going to get cracking on this right now. In the meantime, ben, can you just verify my logic is accurate.
Flags: needinfo?(bhearsum)
Comment #11 is pretty much bang on. The only thing to note is that if we change MOZ_PKG_SPECIAL (or something else) to get a unique build target, we don't need multiple rules or split blobs. We just need to update the mapping in https://github.com/mozilla/build-tools/blob/master/lib/python/release/platforms.py#L23, which corresponds to the "platforms" keys in blobs. Does that make sense?
Flags: needinfo?(bhearsum)
mozharness bits to get android nightlies to work. adds ash equivalent to m-c since we are enabling ash as well
Attachment #8527041 - Flags: review?(hwine)
Attachment #8527039 - Flags: review?(hwine)
Comment on attachment 8527039 [details] [diff] [review]
141121_bug_add_ash_nightlies_apksplit-bbot-cfgs.patch

Review of attachment 8527039 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +1751,4 @@
>      for platform in branch['platforms']:
>          if not platform in PLATFORMS:
>              continue
> +        if platform not in ('android', 'android-api-9'):

We chatted in IRC that this is currently a no-op change.

A note here in the bug on why we're even making the change then would be good.
Attachment #8527039 - Flags: review?(hwine) → review+
Attachment #8527041 - Flags: review?(hwine) → review+
(In reply to Hal Wine [:hwine] (use needinfo) from comment #16)
> Comment on attachment 8527039 [details] [diff] [review]
> 141121_bug_add_ash_nightlies_apksplit-bbot-cfgs.patch
> 
> Review of attachment 8527039 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/mobile_config.py
> @@ +1751,4 @@
> >      for platform in branch['platforms']:
> >          if not platform in PLATFORMS:
> >              continue
> > +        if platform not in ('android', 'android-api-9'):
>

this block affects only api-9 (and obviously the old 'android' as that includes api-9 in it) so I removed it here for clarity sake
turns out we do nightlies for fennec ash builds but we don't do updates. That logic pivots off the create_snippet item and that's set to false for ash.

this affects all android nightly ash builders so we can submit to balrog.

there is one side effect: win64 ash nightlies will now start submitting to balrog. I don't think this is a bad thing though as the other desktop platforms already do anyway.

Why does win64 not submit and the other platforms do you may ask? Because all other platforms avail of mozharness mach and rely on their own configs outside the walls of buildbot-configs. win64 is the only platform that has not made the switch yet.
Attachment #8527960 - Flags: review?(nthomas)
Comment on attachment 8527960 [details] [diff] [review]
141124_split_apk_ash_updates_for_nightlies.patch

some dump_master files if you want to diff against:

http://people.mozilla.org/~jlund/2014-11-24--14:09:19--clean
http://people.mozilla.org/~jlund/2014-11-24--14:37:59--dirty
Comment on attachment 8527960 [details] [diff] [review]
141124_split_apk_ash_updates_for_nightlies.patch

I see 'MOZ_UPDATE_CHANNEL': 'nightly-ash' is now being set in the environment for a bunch of steps, including 'make -f client.mk build'. This means the channel would not have been set properly in old ash nightlies, affecting your testing over the transition to new builders.
Attachment #8527960 - Flags: review?(nthomas) → review+
Comment on attachment 8527960 [details] [diff] [review]
141124_split_apk_ash_updates_for_nightlies.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/4c0f6f990e88
Attachment #8527960 - Flags: checked-in+
I think this is what I need to do. It should leave flame logic as is but will support new mappings for api-9:

if the the build pf is android-api-9, we will submit to a new mapping:

e.g. instead of Fennec-mozilla-central-nightly-latest it will be Fennec-mozilla-central-api-9-nightly-latest

nthomas, how do I test + land this patch?

-- the helpless balrog hacker
Attachment #8528098 - Flags: review?(nthomas)
I had to fix a few busted infra bugs this evening which stole some time away from me. I need to land a patch similar to comment 25 before proceeding.

in other news, it looks like changing the update_channel for testing purposes can only be done pre-compile time by changing the code for MOZ_UPDATE_CHANNEL (bug 792992). this will make testing updates from non-apk-split to one of the new api splits very difficult on m-c.

This fix will not come before tues now FYI
Comment on attachment 8528098 [details] [diff] [review]
141124_split_apk_support_api-9_mapping-tools.patch

hi ben, do you think you would have time to r? this? see https://bugzilla.mozilla.org/show_bug.cgi?id=1080749#c25 for context. If not, feel free to bump it on :)
Attachment #8528098 - Flags: review?(nthomas) → review?(bhearsum)
Comment on attachment 8528098 [details] [diff] [review]
141124_split_apk_support_api-9_mapping-tools.patch

Review of attachment 8528098 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/balrog/submitter/cli.py
@@ +266,5 @@
>  
>          data.update(self._get_update_data(productName, branch, **updateKwargs))
>  
> +        # hack - bug 1055305, 1080749
> +        build_type = self.get_build_type_from_build_target(build_target, self.build_type)

I see what you're doing here, but I would honestly prefer another "quick hack" here. The right fix is much larger in scope than this, and I don't think factoring out the quick hacks into the method above confuses the matter.

With the build_type construction moving back to this method, you can key off of "platform" (which will be android-api-9) instead of adding the fake build target of "Android_arm-eabi-gcc3-api-9".

::: lib/python/release/platforms.py
@@ +22,4 @@
>  # buildbot -> update platform mapping
>  update_platform_map = {
>      'android': ['Android_arm-eabi-gcc3'],
> +    'android-api-9': ['Android_arm-eabi-gcc3-api-9', 'Android_arm-eabi-gcc3'],

...which also means you can remove the fake build target from here.
Attachment #8528098 - Flags: review?(bhearsum) → review-
thanks for the review!

this patch should address those concerns.
Attachment #8528098 - Attachment is obsolete: true
Attachment #8528510 - Flags: review?(bhearsum)
Attachment #8528510 - Flags: review?(bhearsum) → review+
Comment on attachment 8528510 [details] [diff] [review]
141125_split_apk_support_api-9_mapping-tools.patch

Review of attachment 8528510 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/balrog/submitter/cli.py
@@ +248,5 @@
> +            build_type = 'kitkat-%s' % self.build_type
> +        elif platform == 'android-api-9':
> +            # Bug 1080749 - a hack to support api-9 and api-10+ split builds.
> +            # Like 1055305 above, this is a hack to support two builds with same build target that
> +            # require differed't release blobs and rules

will fix typo: s/differed't/different
Attachment #8528510 - Flags: review+ → review?
Attachment #8528510 - Flags: review? → review+
status update:

currently running 3 nightlies in a staging environment
    1 non-split m-c apk (old android build) 
    1 api-9 m-c apk 
    1 api-10 m-c apk

the api split nightlies are being done after so they have newer buildid. They also share the same buildid so I can test the rules working as expected on dev instance of balrog.

once these finish (3 hours), I can manually test update on devices. I have a 2.3 and a 4.0 device.

I also had to change the aus4 request url to point to the balrog dev instance since this is all being done in my staging env: http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/rev/e835b860f129

/me crosses fingers
result of the nightlies:

good news:
    - they were green in staging
    - I can see the updates on the dev node of balrog
    - Testing my tools patch and the new mapping works as expected (api-9 mapping!):
          - snippet: https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-api-9-nightly-20141125204145/builds/Android_arm-eabi-gcc3/en-US
          - buildstep: http://dev-master1.srv.releng.scl3.mozilla.com:8037/builders/Android%20armv7%20API%209%20mozilla-central%20nightly/builds/6/steps/submit_balrog_updates/logs/stdio

bad news:
    - I'm currently blocked on being able to add rules to the dev balrog instance so I can't test if the update.xml request yields the correct release blob for api-9 vs api-10
    - the pre-apk-split nightly build doesn't work on api-9 device (Bug 1100361) so I can't verify if the old android build will apply against the new api-9 split build. This doesn't seem like a big deal since it is already broken for gingerbread nightly users and a clean install is expected
    - after installing pre-apk-split nightly on a 4.0 (api-10+) device, the update failed to download and apply. making a manual request through browser I can see release bits: https://aus4-dev.allizom.org/update/4/Fennec/36.0a1/20141123030204/Android_arm-eabi-gcc3/en-US/nightly/4.4.3/default/default/36.0a1/update.xml but I don't know why it's not working on the device. https://wiki.mozilla.org/Balrog/Testing doesn't seem to apply to fennec
this enables new apk-split on all of trunk. I can't land this yet until I resolve the final pieces in comment 32 but I wanted to get this ready so I can immediately when ready.

see upcoming bbotcustom and mozharness patch to explain motivation for multi_locale_config_platform config item
Attachment #8528859 - Flags: review?(kmoir)
hopefully in-line comment explains reasoning for adding to misc.py. next patch removes the api-9/10 configs I added in mozharness.
Attachment #8528862 - Flags: review?(kmoir)
Comment on attachment 8528859 [details] [diff] [review]
141125_split_apk_ride_trains_and_support_custom_multi_locale_config-bbot-cfgs.patch

Review of attachment 8528859 [details] [diff] [review]:
-----------------------------------------------------------------

The patches look good to me, would be good to see an builder diff too
Attachment #8528859 - Flags: review?(kmoir) → review+
Attachment #8528862 - Flags: review?(kmoir) → review+
Attachment #8528863 - Flags: review?(kmoir) → review+
(In reply to Kim Moir [:kmoir] from comment #36)
> Comment on attachment 8528859 [details] [diff] [review]
> 141125_split_apk_ride_trains_and_support_custom_multi_locale_config-bbot-
> cfgs.patch
> 
> Review of attachment 8528859 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> The patches look good to me, would be good to see an builder diff too

thanks! builder-diff: http://people.mozilla.org/~jlund/141125_split_apk_ride_trains.diff
thanks the builder diff looks good
hey kim, so I am going to try one more thing. I want to try updating from a pre-split 'android' nightly build to one of the new splits. However, the pre-split builds on ash all have the wrong MOZ_UPDATE_CHANNEL.

this patch enables all three builds on ash so I can accomplish the testing I need. it also cherry picks part of my last review so I can avail of not having to worry about the additional mozharness multi locale configs.

build builder-diff: http://people.mozilla.org/~jlund/141125_split_apk_old_apk_ash_cedar.diff

IIUC, this will do the builds and fire a sendchange for 'android' but since I am not enabling the test equivalent of the old 'android' builds, no tests will run for them (which is what I want)
Attachment #8529214 - Flags: review?(kmoir)
You've probably already thought of this, but you could make a try build (or land on a different throwaway twig) a change to the updater URL (just hard-code in UpdaterService.java, or change the update channel), get a 'pre' build from that, then apply your changes on top.

You don't need a build from ash per se; you just need the update channel to agree, right?

That gives you 'pre' and two 'post', all of which are looking at your custom update URL.
(In reply to Richard Newman [:rnewman] from comment #40)
> You've probably already thought of this, but you could make a try build (or
> land on a different throwaway twig) a change to the updater URL (just
> hard-code in UpdaterService.java, or change the update channel), get a 'pre'
> build from that, then apply your changes on top.
> 
> You don't need a build from ash per se; you just need the update channel to
> agree, right?
> 
> That gives you 'pre' and two 'post', all of which are looking at your custom
> update URL.

I was thinking of this but I would need to still kick off two post split 'android' builds anyway as they would need to have newer revs + buildid's. I also thought even if I messed with the update request url in the 'try/twig pre' push, there still might be differences between the code of the try/twig and ash build that might invalidate testing.

Since we have to do new post builds anyway, I figured this would be best. As a bonus, I don't have to mess with UpdaterService.java and I can keep this looking as close to prod as possible.
Attachment #8529214 - Flags: review?(kmoir) → review+
Comment on attachment 8529214 [details] [diff] [review]
141126_split_apk_enable_old_pre_split_android_on_ash_cedar.patch

myself and kmoir touched base on irc.

this patch is the inverse of what I landed. /me fails at hg diff

https://hg.mozilla.org/build/buildbot-configs/rev/27c0d0110ec1
Attachment #8529214 - Flags: checked-in+
Comment on attachment 8528863 [details] [diff] [review]
141125_apk_split_remove_api_level_multi_locale_configs-mozharness.patch

on default https://hg.mozilla.org/build/mozharness/rev/b695efb84faa
Attachment #8528863 - Flags: checked-in+
Comment on attachment 8528862 [details] [diff] [review]
141125_apk_split_support_platform_multi_locale_config_override-bbotcustom.patch

on default: https://hg.mozilla.org/build/buildbotcustom/rev/3dc7615603de
Attachment #8528862 - Flags: checked-in+
Comment on attachment 8528510 [details] [diff] [review]
141125_split_apk_support_api-9_mapping-tools.patch

tested in staging dev instance of balrog

on default: https://hg.mozilla.org/build/tools/rev/0f476664e904
Attachment #8528510 - Flags: checked-in+
this fixes a typo that caused api-9 binaries to being uploaded to api-10 dir. wasn't the worst thing as balrog pointed to things correctly but it would cause confusion.
Attachment #8529311 - Flags: review?(coop)
Attachment #8529311 - Flags: review?(coop) → review+
Attachment #8529311 - Flags: review+
Comment on attachment 8529311 [details] [diff] [review]
141126_fix_api-9_stage_platform.patch

thanks!

on default: https://hg.mozilla.org/build/buildbot-configs/rev/893a4b2e5d5e

r=rail
Attachment #8529311 - Flags: review+
Attachment #8529311 - Flags: checked-in?
Attachment #8529311 - Flags: checked-in? → checked-in+
Comment on attachment 8529311 [details] [diff] [review]
141126_fix_api-9_stage_platform.patch

in production: http://hg.mozilla.org/build/buildbot-configs/rev/78ff1f5d0bb3
so things went well with ash testing yesterday. I was able to manually update to the api-10 nightly from the pre non split build so this *should* be seamless for existing nightly users.

I forgot about enabling tests for apk-split on trunk in my last patch. This appends to the now obsolete r+ I previously received with additions to mobile_config.py

build builder diff: http://people.mozilla.org/~jlund/141127_split_apk_old_apk_trunk-build.diff
test builder diff: http://people.mozilla.org/~jlund/141125_split_apk_old_apk_trunk-test.diff
Attachment #8528859 - Attachment is obsolete: true
Attachment #8529959 - Flags: review?
Attachment #8529959 - Flags: review? → review+
(In reply to Jordan Lund (:jlund) from comment #52)

> so things went well with ash testing yesterday. I was able to manually
> update to the api-10 nightly from the pre non split build so this *should*
> be seamless for existing nightly users.

Does that mean that Gingerbread users will be seamlessly offered the API9 Nightly?
> Does that mean that Gingerbread users will be seamlessly offered the API9
> Nightly?

Jordan confirmed on IRC that this is indeed the case. \o/
due to releases being pushed and two solid days of infra errors, this was blocked from landing today.

since 2.3.* is broken on aurora already and soon to be beta, we should land early next week (monday), and uplift if it sticks on central, now 37, asap.

on the bright side, this gave me time to catch something I missed. It looks like we trigger l10n mobile repacks off our nightly builds. These don't run on ash or show up in tbpl so I missed them. At any rate, I think they will break. We need this patch and then a mozharness (mobile_l10n.py) patch.

unknowns: the pre android split l10n mozconfig (mobile/android/config/mozconfigs/android/l10n-nightly) already had MOZ_DISABLE_GECKOVIEW=1 and --with-android-version=9 which means the api-10 l10n-nightly mozconfig will have both of these too. I am not sure if that's what we want? Maybe this l10n-nighty is out of date or wrong so it be nice to get this verified.

below is an interdiff of the pre android l10n mozconfig vs the split apk ones. 
http://people.mozilla.org/~jlund/l10n_diff-api-9.patch
http://people.mozilla.org/~jlund/l10n_diff-api-10.patch

I'm testing this in staging now. If they pass I'll get a r?
> unknowns: the pre android split l10n mozconfig
> (mobile/android/config/mozconfigs/android/l10n-nightly) already had
> MOZ_DISABLE_GECKOVIEW=1 and --with-android-version=9 which means the api-10
> l10n-nightly mozconfig will have both of these too. I am not sure if that's
> what we want? Maybe this l10n-nighty is out of date or wrong so it be nice
> to get this verified.

I think w-a-v is unnecessary these days for general builds -- my unconstrained mozconfig omits it. The build derives it if it's missing.

If you specify it, it should be =9 and =10 respectively. It *might* work to still target 9 and specify 10+ as the supported range, but I haven't tested that.
these are the mozharness configs that are used by mobile_l10n.py (triggered during nightlies that have pf['l10n_enabled'] true).

They are copies of mozharness/configs/single_locale/mozilla-central_android.py but they swap api-* platform names for things like mozconfigs and upload paths
Attachment #8532214 - Flags: review?(kmoir)
same as before but removed --with-android-version=9 from both split-apk mozconfigs.
Attachment #8530451 - Attachment is obsolete: true
Attachment #8532220 - Flags: review?(rnewman)
coop, this is an interdiff of the un-landed https://bugzilla.mozilla.org/attachment.cgi?id=8529959

it accounts for the m-c version bump and also fixes a bug where esr-31 was not listed in gecko_versions for mobile_config.py. That had the consequence of being included regardless of the min/max versions we pass to things like items_at_least and items_before :(
Attachment #8532241 - Flags: review?(coop)
Attachment #8532241 - Flags: review?(coop) → review+
Attachment #8532214 - Flags: review?(kmoir) → review+
Attachment #8532220 - Flags: review?(rnewman) → review+
okay, I've been able to reproduce the l10n builds and confirmed they work for both api-9 and 10. let's land all of this and get this on trunk now.

I made typo on the mozconfig path for api-9 so this is an interdiff from https://bugzilla.mozilla.org/attachment.cgi?id=8532214&action=edit

I should note that chunk 5 and the sq locale failed. Although it looks to be failing in production already on the pre-split builds.
Attachment #8532615 - Flags: review?(kmoir)
Comment on attachment 8529959 [details] [diff] [review]
141127_split_apk_trunk-bbot-cfgs-2.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/48108ed16077
Attachment #8529959 - Flags: checked-in+
Comment on attachment 8532241 [details] [diff] [review]
141204_split_apk_land_on_trunk_bump_to_37.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/48108ed16077
Attachment #8532241 - Flags: checked-in+
Comment on attachment 8532220 [details] [diff] [review]
141204_split_apk-l10n_nightly_builds-mozilla-central.patch

https://hg.mozilla.org/integration/mozilla-inbound/rev/9066d3ddd908
Attachment #8532220 - Flags: checked-in+
Keywords: leave-open
Attachment #8532615 - Flags: review?(kmoir) → review+
so I need to keep the old 2.3 style as apk split is riding the trains.

the new api-9 test jobs use linux but IIUC, the reftest jobs use a different ec2 slavename (api-10 only uses panda)
Attachment #8532701 - Flags: review?(rail)
Attachment #8532701 - Flags: review?(rail) → review+
Comment on attachment 8532701 [details] [diff] [review]
141205_split_apk-watch-pending-fix-cloud-tools.patch

on default: https://hg.mozilla.org/build/cloud-tools/rev/502e91eb2875
Attachment #8532701 - Flags: checked-in+
Comment on attachment 8532214 [details] [diff] [review]
141204_split_apk_nightly_l10n-mozharness.patch

on default: https://hg.mozilla.org/build/mozharness/rev/5ded87dbee26
merged to production: https://hg.mozilla.org/build/mozharness/rev/3bd893ecb2c9
Attachment #8532214 - Flags: checked-in+
Comment on attachment 8532615 [details] [diff] [review]
141205_split_apk_l10n_nightly-mozconfig_path_fix-mozharness.patch

on default: https://hg.mozilla.org/build/mozharness/rev/5ded87dbee26
merged to production: https://hg.mozilla.org/build/mozharness/rev/3bd893ecb2c9
see https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c82 for context

api-9 is staying as is as it represents the minimum api we support
Attachment #8533550 - Flags: review?(bhearsum)
Attachment #8533550 - Flags: review?(bhearsum) → review+
Comment on attachment 8533551 [details] [diff] [review]
141208_split_apk-new_api_11_split-mh.patch

sorry I missed seeing this was up for review yesterday
Attachment #8533551 - Flags: review?(kmoir) → review+
Comment on attachment 8533550 [details] [diff] [review]
141208_split_apk-new_api_11_split-balrog-tools.patch

on default: https://hg.mozilla.org/build/tools/rev/686789acbc52
Comment on attachment 8533551 [details] [diff] [review]
141208_split_apk-new_api_11_split-mh.patch

on default: https://hg.mozilla.org/build/mozharness/rev/5e29d80fcb57
Attachment #8533551 - Flags: checked-in+
tracking-fennec: 35+ → 37+
I think this work is resolved.

follow up bugs can be done for other release channels
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.