1073772 - Releng work for producing two ARMv7 APKs to target different API ranges

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Richard Newman [:rnewman]]

See meta Bug 1039789.

Comment 1

[Jordan Lund (:jlund)]

one of the first things we will have to decide is which builders we will want to run and where.

e.g. currently we run a number of compile builders for android on mozilla-central ->

opt debug and nonunified:
Android 2.3 {branch} build NightlyBuildFactory
Android 2.3 {branch} non-unified NightlyBuildFactory
Android 2.3 Debug {branch} build NightlyBuildFactory
Android 2.3 Debug {branch} non-unified NightlyBuildFactory
Android 4.2 x86 {branch} build NightlyBuildFactory

nightlies:
Android 2.3 {branch} nightly NightlyBuildFactory
Android 4.2 x86 {branch} nightly NightlyBuildFactory

on mozilla-esr the following still hasn't completed its train ride:
Android 2.3 Armv6 mozilla-esr31 build NightlyBuildFactory
Android 2.3 Armv6 mozilla-esr31 nightly NightlyBuildFactory

after this bug is resolved, what do we want the above list of builders that run to look like? I'm assuming we will be making changes trunk (and try) wide initially and then riding the trains.

Comment 2

[Richard Newman [:rnewman]]

(In reply to Jordan Lund (:jlund) from comment #1)

> Android 2.3 {branch} nightly NightlyBuildFactory
> Android 4.2 x86 {branch} nightly NightlyBuildFactory

What about Android 4.0 ARM, which I see in TBPL?

Is it that these names are conflating testing (which depends on OS revision) with building (which only cares about architecture)?

We would want to do two ARM builds instead of one.

> I'm assuming we will be making changes trunk (and try)
> wide initially and then riding the trains.

Riding the trains is fine by me.

Comment 3

[Jordan Lund (:jlund)]

apologies if I am too verbose. This area is new to me and I want to make sure I am implementing this to the spec you expect.

(In reply to Richard Newman [:rnewman] from comment #2)
> (In reply to Jordan Lund (:jlund) from comment #1)
> 
> > Android 2.3 {branch} nightly NightlyBuildFactory
> > Android 4.2 x86 {branch} nightly NightlyBuildFactory
> 
> What about Android 4.0 ARM, which I see in TBPL?
> 
> Is it that these names are conflating testing (which depends on OS revision)
> with building (which only cares about architecture)?

hm, yes after looking again, this is confusing. So, like we do for windows, I think you're right in that we do one arm build using 2.3. We then test the result of that build against 2.3 (emulator w/ linux instance) and 4.0 (panda).

from this tbpl example: https://tbpl.mozilla.org/?rev=6eb7121059fb you can see that we have a 'B' compile job under 'Android 2.3 Opt', Android 2.3 Debug', and 'Android 4.2 x86 Opt'. In tbpl, we add another line for the 4.0 tests but it does not have a 'B' job at the start (there is only a greyed out 'M'). We do this so we don't overload the 2.3 lines and to segregate 4.0 tests.

for android 4.2 x86, it looks like we do a build for that and trigger only a 'set(4)' test against it

> 
> We would want to do two ARM builds instead of one.

ok so assuming we are on the same page, we want to leave android 4.2 x86 build + tests as is. We would then like to split the android 2.3 opt and debug compile jobs (arm based) up into two. So we would:

for compile jobs:
    replace:
        Android 2.3 {branch} build
        Android 2.3 Debug {branch} build
    with:
        Android 2.3 SDK 9 {branch} build
        Android 2.3 SDK 10+ {branch} build
        Android 2.3 Debug SDK 9 {branch} build
        Android 2.3 SDK 10+ {branch} build

then those 4 new builds will be used against 2.3 and 4.0 tests. So really we will have 8 lines for tests:
    Android 2.3 SDK 9 {branch} tests 
    Android 2.3 SDK 10+ {branch} tests
    Android 2.3 Debug SDK 9 {branch} tests
    Android 2.3 SDK 10+ {branch} tests
    Android 4.0 Opt SDK 9 {branch} tests
    Android 4.0 Opt SDK 10+ {branch} tests
    Android 4.0 Debug SDK 9 {branch} tests
    Android 4.0 Debug SDK 10+ {branch} tests

So with that I see two big questions:
    1) how can we divide tests amongst the above 8 test lines so we do not run redundant tests?
    2) with regards to 'Android 2.3 non-unified' and 'Android 2.3 Debug non-unified' (these builds do not trigger tests), do we need to split them up too or can we do them against just say the SDK 10+ builder?

where do we go from here? what I propose we do right away is:
    1) split 'Android 2.3 Debug build' job into two within a staging environment until they at least compile and create APKs as expected
    2) Once we have that working and some tests running against it, we add the two new split builders to Cedar and run them simultaneously with their combined sdk 9 + 10 counterpart to ensure we are not regressing.
   3) iterate until we are not duplicating tests without need
   4) replace trunk and try builders + tests
   5) ride trains

Comment 4

[Jordan Lund (:jlund)]

> 
> for compile jobs:
>     replace:
>         Android 2.3 {branch} build
>         Android 2.3 Debug {branch} build
>     with:
>         Android 2.3 SDK 9 {branch} build
>         Android 2.3 SDK 10+ {branch} build
>         Android 2.3 Debug SDK 9 {branch} build
>         Android 2.3 SDK 10+ {branch} build
> 
> then those 4 new builds will be used against 2.3 and 4.0 tests. So really we
> will have 8 lines for tests:
>     Android 2.3 SDK 9 {branch} tests 
>     Android 2.3 SDK 10+ {branch} tests
>     Android 2.3 Debug SDK 9 {branch} tests
>     Android 2.3 SDK 10+ {branch} tests
>     Android 4.0 Opt SDK 9 {branch} tests
>     Android 4.0 Opt SDK 10+ {branch} tests
>     Android 4.0 Debug SDK 9 {branch} tests
>     Android 4.0 Debug SDK 10+ {branch} tests


I think there is a way to simplify this. I was confused as I thought API 10+ still accounted for 2.3 so we would need to test and build against that too. this comment[1] suggests that's not necessarily the case. So instead we we could have:

replace:
    Android 2.3 {branch} build
    Android 2.3 Debug {branch} build
with:
    Android 2.3 API 9 {branch} build
    Android 4.0 API 10+ {branch} build
    Android 2.3 Debug API 9 {branch} build
    Android 4.0 Debug API 10+ {branch} build

then with tests we would still have 4 groups not 8:
    Android 2.3 API 9 {branch} tests 
    Android 2.3 Debug API 9 {branch} tests
    Android 4.0 Opt API 10+ {branch} tests
    Android 4.0 Debug API 10+ {branch} tests

rnewman:
    1) does the above sound right? Is this what you are hoping for?
    2) do you know if we should do nonunified builds against a '2.3 API 9' build or a '4.0 API 10+'. It be nice if we didn't have to do both.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1039789#c15

Comment 5

[Richard Newman [:rnewman]]

(In reply to Jordan Lund (:jlund) from comment #4)

> >     Android 2.3 SDK 9 {branch} tests 
> >     Android 2.3 SDK 10+ {branch} tests

The reason you can simplify below is that 2.3 can *only* run API9, and >2.3 can *only* run 10+, so we couldn't do the above even if we wanted to.


>     Android 2.3 API 9 {branch} build
>     Android 4.0 API 10+ {branch} build
>     Android 2.3 Debug API 9 {branch} build
>     Android 4.0 Debug API 10+ {branch} build

I think these names are pretty confusing; "4.0" has nothing to do with building. I think what you mean is

  armv7 API 9-9 {debug,release}
  armv7 API 10-* {debug, release}

which is what I would expect, in addition to the existing:

  x86 API 9-* {debug,release]

So we would do 6 total builds instead of the 4 we do now.

If we currently do different builds for 2.3 and 4.0 (i.e., we do 6 builds now), but we use exactly the same tools and settings, then we're doing double the work for no reason. The Android OS version isn't part of the build process (with the obvious exception of the SDK ranges we're discussing adding here).


> then with tests we would still have 4 groups not 8:
>     Android 2.3 API 9 {branch} tests 
>     Android 2.3 Debug API 9 {branch} tests
>     Android 4.0 Opt API 10+ {branch} tests
>     Android 4.0 Debug API 10+ {branch} tests

Correct, plus

  Android 4.* {opt, debug} x86 API 9-*.


>     2) do you know if we should do nonunified builds against a '2.3 API 9'
> build or a '4.0 API 10+'. It be nice if we didn't have to do both.

What is a nonunified build?

Comment 6

[Richard Newman [:rnewman]]

Just to avoid you grubbing through other bugs, the mozconfig changes:

10+ (current ARMv7 config named "android"):

ac_add_options --with-android-min-sdk=10

9 (clone of current ARMv7 "android", name it "android-9-constrained"):

ac_add_options --with-android-min-sdk=9
ac_add_options --with-android-max-sdk=9
ac_add_options --enable-android-resource-constrained
ac_add_options --disable-android-include-fonts

If those fail to build, let me know! There are a couple of changes that Lucas is about to land that might be required to unbreak the constrained build, 'cos this doesn't currently get tbpl love.

Comment 7

[Jordan Lund (:jlund)]

> The reason you can simplify below is that 2.3 can *only* run API9, and >2.3
> can *only* run 10+, so we couldn't do the above even if we wanted to.

thanks, as mentioned in IRC, this helps. I think I'm caught up now with how things are done.

> 
> 
> >     Android 2.3 API 9 {branch} build
> >     Android 4.0 API 10+ {branch} build
> >     Android 2.3 Debug API 9 {branch} build
> >     Android 4.0 Debug API 10+ {branch} build
> 
> I think these names are pretty confusing; "4.0" has nothing to do with
> building. I think what you mean is
> 
>   armv7 API 9-9 {debug,release}
>   armv7 API 10-* {debug, release}
> 
> which is what I would expect, in addition to the existing:
> 
>   x86 API 9-* {debug,release]
> 
> So we would do 6 total builds instead of the 4 we do now.

sgtm.

edmorely: while I reconfig our infra for this, do you recommend or have a preference on builder names? e.g. can I use things like '*' or will that confuse tbpl regex? I think, either way, we should remove the '2.3' from the build compile job names as it seems to be meaningless.

I can open up a tbpl + treeherder bug for that work if you like.

Comment 8

[Richard Newman [:rnewman]]

(In reply to Jordan Lund (:jlund) from comment #7)
> e.g. can I use things like '*' or will that
> confuse tbpl regex?

I'm far from wedded to "*", fwiw; you can use anything you like to indicate "ten and up" or "nine and up".

Comment 9

[Jordan Lund (:jlund)]

I'm having an issue building this in staging. Assuming its one of my only hurdles, I should have a patch by EOD tomorrow for an initial impl of this.

Comment 10

[Richard Newman [:rnewman]]

Thanks for the haste, jlund!

Let me know if the hurdles are anything I can help with. (Particularly compile or package errors!)

Comment 11

[Ed Morley [:emorley]]

(In reply to Jordan Lund (:jlund) from comment #7)
> edmorely: while I reconfig our infra for this, do you recommend or have a
> preference on builder names? e.g. can I use things like '*' or will that
> confuse tbpl regex? I think, either way, we should remove the '2.3' from the
> build compile job names as it seems to be meaningless.
...
>   armv7 API 9-9 {debug,release}
>   armv7 API 10-* {debug, release}
> 
> which is what I would expect, in addition to the existing:
> 
>   x86 API 9-* {debug,release]

The current buildernames are of form:
1) Android 2.3 mozilla-central {build,non-unified,nightly}
2) Android 2.3 Debug mozilla-central {build,non-unified}
3) Android 4.2 x86 mozilla-central {build,nightly}

Things I don't like about these:
* As you say, the "2.3" and "4.2" are redundant, we should omit them.
* The "Debug" is in the wrong location compared to all other debug jobs - it's typically after the repository name.

Re your proposal:
* I think "10+" is preferable to "10-*" since it's easier to grok and avoids the potential issues with the asterisk. 
* "9-9" should just be "9".
* Are we doing nightly/non-unified of everything? The x86 builds currently don't do non-unified. I think we could get away with not doing non-unified for API-9 too (given that it only affects C++ parts of the build, for which I imagine there will be few differences between the build types).
* Why 9+ for x86 and not split into two as well?

Taking the above into account, what about:

1)
  a) Android armv7 API-9 mozilla-central {build,nightly}
  b) Android armv7 API-10+ mozilla-central {build,non-unified,nightly}
2)
  a) Android armv7 API-9 mozilla-central debug {build,nightly}
  b) Android armv7 API-10+ mozilla-central debug {build,non-unified,nightly}
3)   Android x86 API-9+ mozilla-central {build,nightly}  [though should we split this into two, too?]


> I can open up a tbpl + treeherder bug for that work if you like.

The current TBPL/Treeherder regexes are at:
https://hg.mozilla.org/webtools/tbpl/file/default/js/Data.js#l589
https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py#L176

I'll file bugs in readiness for the changes :-)

Comment 12

[Ed Morley [:emorley]]

Sorry, ignore the nightly debug variants, I meant to remove that after copy pasta-ing.

Comment 13

[Ed Morley [:emorley]]

Also, are we running tests on all of these?
If so, what will the test buildernames be?

Comment 14

[Richard Newman [:rnewman]]

(In reply to Ed Morley [:edmorley] from comment #11)

> * Are we doing nightly/non-unified of everything? The x86 builds currently
> don't do non-unified. I think we could get away with not doing non-unified
> for API-9 too (given that it only affects C++ parts of the build, for which
> I imagine there will be few differences between the build types).

I don't know what a non-unified build is, sorry. (And MDN is no help.)

I'm assuming it's some kind of perf- or build-time related source transformation.

If so, I think you're asking if we need to do both "fast" and "correct" builds for every 'slice', to determine if there are problems in that transformation.

I don't think we do. The main ARMv7 build is likely to catch any bugs that the others don't.


> * Why 9+ for x86 and not split into two as well?

It's not worth the build effort to please those five users. We only really care about API15+ for x86 (Bug 1062537).

> 
> Taking the above into account, what about:
> 
> 1)
>   a) Android armv7 API-9 mozilla-central {build,nightly}
> ...

Yes, though I'm inclined to include the word "constrained" in the API9 build description, somewhere. They aren't just built for API9, they exclude several MB of stuff and some device configurations -- targeting "resource constrained" devices. That'll help chasing down some kinds of bugs.


> I'll file bugs in readiness for the changes :-)

Thanks!

Comment 15

[Richard Newman [:rnewman]]

(In reply to Ed Morley [:edmorley] from comment #13)
> Also, are we running tests on all of these?
> If so, what will the test buildernames be?

The 2.3 tests will run on the "9" build. The 4.0 tests will run on the "10+" build. The x86 tests will be unaffected.

There should be no need to change names, or anything apart from which APK the testers use -- 2.3 can *only* run the v9 build, and 4.0 can *only* run the 10+ build, so no other configuration makes sense.

Comment 16

[Ed Morley [:emorley]]

(In reply to Richard Newman [:rnewman] from comment #15)
> 2.3 can *only* run the v9 build, and 4.0 can *only* run
> the 10+ build, so no other configuration makes sense.

Ah of course, I didn't stop to think about that properly!

(In reply to Richard Newman [:rnewman] from comment #14)
> (In reply to Ed Morley [:edmorley] from comment #11)
> 
> > * Are we doing nightly/non-unified of everything? The x86 builds currently
> > don't do non-unified. I think we could get away with not doing non-unified
> > for API-9 too (given that it only affects C++ parts of the build, for which
> > I imagine there will be few differences between the build types).
> 
> I don't know what a non-unified build is, sorry. (And MDN is no help.)
> 
> I'm assuming it's some kind of perf- or build-time related source
> transformation.

Yeah - concatenation of C++ sources to reduce build times (https://wiki.mozilla.org/Platform/Porting_to_unified_sources), that may mask breakage in non-concatenated builds (eg due to improper includes or forward declaration) - which means we choose to run occasional (every few hours) non-concatenated builds on trunk, to avoid us having a surprise when we merge to aurora/beta/release, since we don't concatenate there.

> If so, I think you're asking if we need to do both "fast" and "correct"
> builds for every 'slice', to determine if there are problems in that
> transformation.

Spot on

> I don't think we do. The main ARMv7 build is likely to catch any bugs that
> the others don't.

Sgtm :-)

Comment 17

[Jordan Lund (:jlund)]

Attachment: 20141006_1073772_split_apk_by_api.bbotcfgs.patch

Comment 18

[Jordan Lund (:jlund)]

Attachment: 20141006_1073772_split_apk_by_api.m-c.patch

Comment 19

[Jordan Lund (:jlund)]

Comment on attachment 8500854 [details] [diff] [review]
20141006_1073772_split_apk_by_api.bbotcfgs.patch

Review of attachment 8500854 [details] [diff] [review]:
-----------------------------------------------------------------

this patch adds two new android variants. It keeps 'android' (used for opt, nonunified, and nightly) and creates 'android-api-9' and 'android-api-10'.

It will only enable them on cedar.

They will show up on tbpl as:
   'Android armv7 API 9 constrained %(branch)s'
   'Android armv7 API 10+ %(branch)s'

They use the following mozconfigs:
    https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c18

I will do the debug equivalents next (and will fix debug naming convention mentioned by edmorely)

I've got the 10+ to build and green through staging automation. I've got to where I can get the 9 constrained build to compile fine but the 'make package' step fails: http://people.mozilla.org/~jlund/20141006_1073772_split_apk_by_api.fails_make_package.log

Once I can green up both builds, I'll work on patching the tests so they are triggered off the respective build. For tests, we will need a substantial patch even though we are not changing the overall test name or the amount of tests we are doing. This is due to the way that tests are triggered from builds and how we configure that in buildbot.

::: mozilla/config.py
@@ +1424,5 @@
> +            'create_partial': False,
> +            'slaves': SLAVES['mock'],
> +            'platform_objdir': OBJDIR,
> +            # XXX JLUND TODO differentiate update_platform
> +            'update_platform': 'Android_arm-eabi-gcc3',

IIUC, we will no longer need to set this. It is used for our old update mechanism via snippets. However, we will have figure out how to support for balrog (new update server).

This isn't needed for cedar as cedar does not run nightlies.

Comment 20

[Richard Newman [:rnewman]]

     [aapt] /builds/slave/m-cen-and-api-9-ntly-000000000/build/obj-firefox/mobile/android/geckoview_library/res/values-large-v11/styles.xml:89: error: Error: No resource found that matches the given name (at 'android:background' with value '@drawable/new_tablet_action_bar_button').

Should be Bug 1073474, which merged to m-c on 2014-10-04.

Lucas will know whether this is a new regression.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Can we drop the "constrained" from the API9 job names? Seems extraneous for it to be included in the buildbot name (since we're not running "unconstrained" API9 builds as well). On IRC, rnewman seemed OK with having it included in the tooltip description of the job on TBPL/TH.

Comment 22

[Richard Newman [:rnewman]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #21)
> Can we drop the "constrained" from the API9 job names? Seems extraneous for
> it to be included in the buildbot name (since we're not running
> "unconstrained" API9 builds as well). On IRC, rnewman seemed OK with having
> it included in the tooltip description of the job on TBPL/TH.

Yes.

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Also, what happens if and when we decide to add tests running on KitKat? I'm thinking the test name should still contain the Android version it's running on so we don't have to hack around it later.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

Chatting with rnewman on IRC, it's possible we'll be running tests on these API10+ builds on newer Android versions down the road as well.

So I propose that we add 4.0 to the API10+ test names, which makes adding support for jobs running on newer versions trivial down the road. No need to do that for API9 since they'll only ever run on 2.3, though.

So to be clear, the new jobs would be:

Android armv7 API 9 %(branch)s %(job) (builds and tests)
Android armv7 API 9 %(branch)s debug %(job)

Android armv7 API 10+ %(branch)s %(build) (since the builds aren't tied to version, just API)
Android 4.0 armv7 API 10+ %(branch)s %(test)
Android armv7 API 10+ %(branch)s debug %(build)
Android 4.0 armv7 API 10+ %(branch)s debug %(test)

Does that make sense?

Comment 25

[Jordan Lund (:jlund)]

> So to be clear, the new jobs would be:
> 
> Android armv7 API 9 %(branch)s %(job) (builds and tests)
> Android armv7 API 9 %(branch)s debug %(job)
> 
> Android armv7 API 10+ %(branch)s %(build) (since the builds aren't tied to
> version, just API)
> Android 4.0 armv7 API 10+ %(branch)s %(test)
> Android armv7 API 10+ %(branch)s debug %(build)
> Android 4.0 armv7 API 10+ %(branch)s debug %(test)
> 
> Does that make sense?

sgtm. I'll reconfigure accordingly.

Comment 26

[Jordan Lund (:jlund)]

> sgtm. I'll reconfigure accordingly.

while I do that, I'll also just update my status so hopefully we can tackle my current hurdle.

update:

I've added the non-unified and debug builds

the following builds are green:
    Android armv7 API 10+ mozilla-central build
    Android armv7 API 10+ mozilla-central debug build
    Android armv7 API 10+ mozilla-central debug non-unified
    Android armv7 API 10+ mozilla-central non-unified

the following are red:
    Android armv7 API 10+ mozilla-central nightly
    Android armv7 API 9 constrained mozilla-central nightly
    Android armv7 API 9 constrained mozilla-central build
    Android armv7 API 9 constrained mozilla-central debug build


For the nightlies that are failing I have filed a separete bug for that: bug 1080749. We don't need nightly jobs right away as we can get the other builds + tests in automation first.

for the API 9 builds I am still hitting failures with 'make package'. I have confirmed that I have the the patches applied from bug mentioned in comment 20. This[1] is the repo I am using.

Here[2] is the failure and the respective mozconfig[3] used against the build. fwiw, it seemed to compile successfully

lucas: is this a new issue or one that has been expected to be resolved?

[1] http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/
[2] http://people.mozilla.org/~jlund/splitapk_api9_fail_package.log
[3] http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/file/408e7f782e0f/mobile/android/config/mozconfigs/android-api-9-constrained/debug

Comment 27

[Jordan Lund (:jlund)]

> for the API 9 builds I am still hitting failures with 'make package'. I have
> confirmed that I have the the patches applied from bug mentioned in comment
> 20. This[1] is the repo I am using.
> 
> Here[2] is the failure and the respective mozconfig[3] used against the
> build. fwiw, it seemed to compile successfully
> 
> lucas: is this a new issue or one that has been expected to be resolved?
> 
> [1] http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/
> [2] http://people.mozilla.org/~jlund/splitapk_api9_fail_package.log
> [3]
> http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/file/
> 408e7f782e0f/mobile/android/config/mozconfigs/android-api-9-constrained/debug

forgot to add a needinfo for lucas

Comment 28

[Jordan Lund (:jlund)]

Attachment: 20141014_1073772_split_apk_by_api-cedar-only.bbotcfgs.patch

this patch does modifies the following build builders:

(from builder_list.py)
 Android 4.2 x86 larch build NightlyBuildFactory
-Android 2.3 cedar build NightlyBuildFactory
-Android 2.3 cedar non-unified NightlyBuildFactory
-Android 2.3 Debug cedar build NightlyBuildFactory
-Android 2.3 Debug cedar non-unified NightlyBuildFactory
+Android armv7 API 10+ cedar build NightlyBuildFactory
+Android armv7 API 10+ cedar non-unified NightlyBuildFactory
+Android armv7 API 9 cedar build NightlyBuildFactory
+Android armv7 API 10+ cedar debug build NightlyBuildFactory
+Android armv7 API 10+ cedar debug non-unified NightlyBuildFactory
+Android armv7 API 9 cedar debug build NightlyBuildFactory
 Android 4.2 x86 cedar build NightlyBuildFactory


this patch does modifies the following test builders:


(from builder_list.py. this doesn't actually change the amount of test builders)
 Android 4.2 x86 Emulator mozilla-esr31 opt test androidx86-set-4 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-3 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-4 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-5 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-6 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-3 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-4 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-5 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-6 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-7 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-8 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-9 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-10 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-11 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-12 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-13 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-14 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-15 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test plain-reftest-16 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test crashtest-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test crashtest-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-3 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-4 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-5 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-6 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-7 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-8 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-9 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-10 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-11 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-12 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-13 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-14 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-15 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test mochitest-16 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test robocop-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test robocop-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test robocop-3 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test robocop-4 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test xpcshell-1 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test xpcshell-2 ScriptFactory
+Android 2.3 Armv6 Emulator mozilla-esr31 opt test xpcshell-3 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-1 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-2 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-3 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-4 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-5 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-6 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-7 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-8 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-9 ScriptFactory
 Android 2.3 Emulator mozilla-release opt test mochitest-10 ScriptFactory
@@ -15721,138 +15721,138 @@ Android 2.3 Emulator b2g-inbound opt tes
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-10 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-11 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-12 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-13 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-14 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-15 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test plain-reftest-16 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test crashtest-1 ScriptFactory
 Android 2.3 Emulator b2g-inbound opt test crashtest-2 ScriptFactory
 Android 4.2 x86 Emulator b2g-inbound opt test androidx86-set-4 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-1 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-2 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-3 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-4 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-5 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-6 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-7 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-8 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-9 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-10 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-11 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-12 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-13 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-14 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-15 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-16 ScriptFactory
-Android 2.3 Emulator cedar opt test robocop-1 ScriptFactory
-Android 2.3 Emulator cedar opt test robocop-2 ScriptFactory
-Android 2.3 Emulator cedar opt test robocop-3 ScriptFactory
-Android 2.3 Emulator cedar opt test robocop-4 ScriptFactory
-Android 2.3 Emulator cedar opt test xpcshell-1 ScriptFactory
-Android 2.3 Emulator cedar opt test xpcshell-2 ScriptFactory
-Android 2.3 Emulator cedar opt test xpcshell-3 ScriptFactory
-Android 2.3 Emulator cedar opt test mochitest-gl ScriptFactory
-Android 4.0 Panda cedar talos remote-tspaint ScriptFactory
-Android 4.0 Panda cedar talos remote-tsvgx ScriptFactory
-Android 4.0 Panda cedar talos remote-trobopan ScriptFactory
-Android 4.0 Panda cedar talos remote-tp4m_nochrome ScriptFactory
-Android 4.0 Panda cedar talos remote-troboprovider ScriptFactory
-Android 4.0 Panda cedar talos remote-trobocheck2 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-1 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-2 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-3 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-4 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-5 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-6 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-7 ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-8 ScriptFactory
-Android 4.0 Panda cedar opt test robocop-1 ScriptFactory
-Android 4.0 Panda cedar opt test robocop-2 ScriptFactory
-Android 4.0 Panda cedar opt test robocop-3 ScriptFactory
-Android 4.0 Panda cedar opt test robocop-4 ScriptFactory
-Android 4.0 Panda cedar opt test robocop-5 ScriptFactory
-Android 4.0 Panda cedar opt test jsreftest-1 ScriptFactory
-Android 4.0 Panda cedar opt test jsreftest-2 ScriptFactory
-Android 4.0 Panda cedar opt test jsreftest-3 ScriptFactory
-Android 4.0 Panda cedar opt test crashtest ScriptFactory
-Android 4.0 Panda cedar opt test mochitest-gl ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-1 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-2 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-3 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-4 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-5 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-6 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-7 ScriptFactory
-Android 4.0 Panda cedar opt test plain-reftest-8 ScriptFactory
-Android 4.0 Panda cedar opt test xpcshell ScriptFactory
-Android 4.0 Panda cedar opt test jittest ScriptFactory
-Android 4.0 Panda cedar opt test cppunit ScriptFactory
-Android 4.0 Panda cedar opt test instrumentation-browser ScriptFactory
-Android 4.0 Panda cedar opt test instrumentation-background ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-1 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-2 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-3 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-4 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-5 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-6 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-7 ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-8 ScriptFactory
-Android 4.0 Panda cedar debug test robocop-1 ScriptFactory
-Android 4.0 Panda cedar debug test robocop-2 ScriptFactory
-Android 4.0 Panda cedar debug test robocop-3 ScriptFactory
-Android 4.0 Panda cedar debug test robocop-4 ScriptFactory
-Android 4.0 Panda cedar debug test robocop-5 ScriptFactory
-Android 4.0 Panda cedar debug test jsreftest-1 ScriptFactory
-Android 4.0 Panda cedar debug test jsreftest-2 ScriptFactory
-Android 4.0 Panda cedar debug test jsreftest-3 ScriptFactory
-Android 4.0 Panda cedar debug test crashtest ScriptFactory
-Android 4.0 Panda cedar debug test mochitest-gl ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-1 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-2 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-3 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-4 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-5 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-6 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-7 ScriptFactory
-Android 4.0 Panda cedar debug test plain-reftest-8 ScriptFactory
-Android 4.0 Panda cedar debug test xpcshell ScriptFactory
-Android 4.0 Panda cedar debug test jittest ScriptFactory
-Android 4.0 Panda cedar debug test cppunit ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-1 ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-2 ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-3 ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-4 ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-5 ScriptFactory
-Android 2.3 Emulator cedar opt test jsreftest-6 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-1 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-2 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-3 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-4 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-5 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-6 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-7 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-8 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-9 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-10 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-11 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-12 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-13 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-14 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-15 ScriptFactory
-Android 2.3 Emulator cedar opt test plain-reftest-16 ScriptFactory
-Android 2.3 Emulator cedar opt test crashtest-1 ScriptFactory
-Android 2.3 Emulator cedar opt test crashtest-2 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-1 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-2 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-3 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-4 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-5 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-6 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-7 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-8 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-9 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-10 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-11 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-12 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-13 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-14 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-15 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-16 ScriptFactory
+Android armv7 API 9 cedar opt test robocop-1 ScriptFactory
+Android armv7 API 9 cedar opt test robocop-2 ScriptFactory
+Android armv7 API 9 cedar opt test robocop-3 ScriptFactory
+Android armv7 API 9 cedar opt test robocop-4 ScriptFactory
+Android armv7 API 9 cedar opt test xpcshell-1 ScriptFactory
+Android armv7 API 9 cedar opt test xpcshell-2 ScriptFactory
+Android armv7 API 9 cedar opt test xpcshell-3 ScriptFactory
+Android armv7 API 9 cedar opt test mochitest-gl ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-1 ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-2 ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-3 ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-4 ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-5 ScriptFactory
+Android armv7 API 9 cedar opt test jsreftest-6 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-1 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-2 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-3 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-4 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-5 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-6 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-7 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-8 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-9 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-10 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-11 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-12 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-13 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-14 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-15 ScriptFactory
+Android armv7 API 9 cedar opt test plain-reftest-16 ScriptFactory
+Android armv7 API 9 cedar opt test crashtest-1 ScriptFactory
+Android armv7 API 9 cedar opt test crashtest-2 ScriptFactory
 Android 4.2 x86 Emulator cedar opt test androidx86-set-4 ScriptFactory
 Android 4.2 x86 Emulator cedar opt test androidx86-set-1 ScriptFactory
 Android 4.2 x86 Emulator cedar opt test androidx86-set-2 ScriptFactory
 Android 4.2 x86 Emulator cedar opt test androidx86-set-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-tspaint ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-tsvgx ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-trobopan ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-tp4m_nochrome ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-troboprovider ScriptFactory
+Android 4.0 armv7 API 10+ cedar talos remote-trobocheck2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-6 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-7 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-8 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test robocop-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test robocop-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test robocop-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test robocop-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test robocop-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test jsreftest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test jsreftest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test jsreftest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test crashtest ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test mochitest-gl ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-6 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-7 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test plain-reftest-8 ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test xpcshell ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test jittest ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test cppunit ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test instrumentation-browser ScriptFactory
+Android 4.0 armv7 API 10+ cedar opt test instrumentation-background ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-6 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-7 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-8 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test robocop-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test robocop-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test robocop-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test robocop-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test robocop-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test jsreftest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test jsreftest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test jsreftest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test crashtest ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test mochitest-gl ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-1 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-2 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-3 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-4 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-5 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-6 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-7 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test plain-reftest-8 ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test xpcshell ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test jittest ScriptFactory
+Android 4.0 armv7 API 10+ cedar debug test cppunit ScriptFactory
 Android 2.3 Emulator graphics opt test mochitest-1 ScriptFactory


essentially, I am only modifying cedar for now but adding patches in place across all our branches to support the apksplit once we are ready to flip the switch.

The builder names that will show on tbpl should match the finalized requests in comment 24.

'Android armv7 API 9 cedar debug build NightlyBuildFactory' and 'Android armv7 API 9 cedar debug build NightlyBuildFactory' are still failing the package target outlined in comment 26 so this patch should only land after that is fixed.

As far as the tests go, my plan is to iterate on cedar with ateam and fix up any individual job that needs to be tweaked or removed in accordance to https://bugzilla.mozilla.org/show_bug.cgi?id=1039789#c15

mozilla-central patch incoming.

cedar doesn't do nightlies so we don't have to worry about bug 1080749 but I will be switching focus to that bug once this has landed.

kmoir, I can upload dump_master diffs if it helps. Also, feel free to bounce me to someone else if you are not a good candidate for reviewing either the build or test part. thanks!

Comment 29

[Jordan Lund (:jlund)]

Attachment: 20141014_1073772_split_apk_by_api.m-c.patch

accompanying m-c mozconfig patch to comment 28 buildbot-config attachment

Comment 30

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #28)
> Created attachment 8505257 [details] [diff] [review]
> 20141014_1073772_split_apk_by_api-cedar-only.bbotcfgs.patch
> 
> this patch does modifies the following build builders:
> 
> (from builder_list.py)
>  Android 4.2 x86 larch build NightlyBuildFactory
> -Android 2.3 cedar build NightlyBuildFactory
> -Android 2.3 cedar non-unified NightlyBuildFactory
> -Android 2.3 Debug cedar build NightlyBuildFactory
> -Android 2.3 Debug cedar non-unified NightlyBuildFactory
> +Android armv7 API 10+ cedar build NightlyBuildFactory
> +Android armv7 API 10+ cedar non-unified NightlyBuildFactory
> +Android armv7 API 9 cedar build NightlyBuildFactory
> +Android armv7 API 10+ cedar debug build NightlyBuildFactory
> +Android armv7 API 10+ cedar debug non-unified NightlyBuildFactory
> +Android armv7 API 9 cedar debug build NightlyBuildFactory
>  Android 4.2 x86 cedar build NightlyBuildFactory
> 
> 
> this patch does modifies the following test builders:
> 
> 
> (from builder_list.py. this doesn't actually change the amount of test
> builders)
>  Android 4.2 x86 Emulator mozilla-esr31 opt test androidx86-set-4
> ScriptFactory
> +Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-1 ScriptFactory
> +Android 2.3 Armv6 Emulator mozilla-esr31 opt test jsreftest-2 ScriptFactory

I shouldn't have added the '+Android 2.3 Armv6 Emulator mozilla-esr31' diff lines. builder_list moved some of the order to these builders so there is an accompanying - line for each of these. armv6 should be untouched by this patch

Comment 31

[Lucas Rocha (:lucasr)]

(In reply to Jordan Lund (:jlund) from comment #26)
> > sgtm. I'll reconfigure accordingly.
> 
> while I do that, I'll also just update my status so hopefully we can tackle
> my current hurdle.
> 
> update:
> 
> I've added the non-unified and debug builds
> 
> the following builds are green:
>     Android armv7 API 10+ mozilla-central build
>     Android armv7 API 10+ mozilla-central debug build
>     Android armv7 API 10+ mozilla-central debug non-unified
>     Android armv7 API 10+ mozilla-central non-unified
> 
> the following are red:
>     Android armv7 API 10+ mozilla-central nightly
>     Android armv7 API 9 constrained mozilla-central nightly
>     Android armv7 API 9 constrained mozilla-central build
>     Android armv7 API 9 constrained mozilla-central debug build
> 
> 
> For the nightlies that are failing I have filed a separete bug for that: bug
> 1080749. We don't need nightly jobs right away as we can get the other
> builds + tests in automation first.
> 
> for the API 9 builds I am still hitting failures with 'make package'. I have
> confirmed that I have the the patches applied from bug mentioned in comment
> 20. This[1] is the repo I am using.
> 
> Here[2] is the failure and the respective mozconfig[3] used against the
> build. fwiw, it seemed to compile successfully
> 
> lucas: is this a new issue or one that has been expected to be resolved?
> 
> [1] http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/
> [2] http://people.mozilla.org/~jlund/splitapk_api9_fail_package.log
> [3]
> http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/file/
> 408e7f782e0f/mobile/android/config/mozconfigs/android-api-9-constrained/debug

It seems the geckoview sample app ignores the newtabletui flag. Let me see if I can find a workaround.

Comment 32

[Kim Moir [:kmoir] ET]

Comment on attachment 8505257 [details] [diff] [review]
20141014_1073772_split_apk_by_api-cedar-only.bbotcfgs.patch

lgtm

Comment 33

[Kim Moir [:kmoir] ET]

Comment on attachment 8505259 [details] [diff] [review]
20141014_1073772_split_apk_by_api.m-c.patch

So I'm sure this works because from your bug comments you indicate that you have green builds in staging. But I've never touched mozconfigs before so it would be better to get someone familiar with this area to review.

Comment 34

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

re comment 26: I just built locally and after fixing an unrelated compile error to the mentioned issues (fix is landed on fx-team, the patch in bug 1073474 comment 11), did not have the issues mentioned in that bug. I looked in <obj-dir>/mobile/android/geckoview_library/res/drawable/new_tablet_site_security_level.xml and the stub existed - I'm not sure why this error would occur on the most recent fx-team pull.

Jordan, do you mind trying a build again (after waiting for the fix in bug 1073474 to land on m-c)?

Comment 35

[Jordan Lund (:jlund)]

(In reply to Michael Comella (:mcomella) from comment #34)
> re comment 26: I just built locally and after fixing an unrelated compile
> error to the mentioned issues (fix is landed on fx-team, the patch in bug
> 1073474 comment 11), did not have the issues mentioned in that bug. I looked
> in
> <obj-dir>/mobile/android/geckoview_library/res/drawable/
> new_tablet_site_security_level.xml and the stub existed - I'm not sure why
> this error would occur on the most recent fx-team pull.
> 
> Jordan, do you mind trying a build again (after waiting for the fix in bug
> 1073474 to land on m-c)?

sounds good, will test again. the error I hit in comment 6 was from over a week ago. The problem could be resolved.

fwiw my testing was based off my user repo at this rev: http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/rev/e88e4a03bedb (its last m-c merge was oct 7th, 10 days ago)

Comment 36

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Jordan Lund (:jlund) from comment #35)
> fwiw my testing was based off my user repo at this rev:
> http://hg.mozilla.org/users/jlund_mozilla.com/mozilla-central/rev/
> e88e4a03bedb (its last m-c merge was oct 7th, 10 days ago)

Looks like the bug I'd expect to exist exists in that revision so it's looking hopeful! :)

Comment 37

[Jordan Lund (:jlund)]

Comment on attachment 8505259 [details] [diff] [review]
20141014_1073772_split_apk_by_api.m-c.patch

catlee, kim would prefer if I poked someone more familiar with our android mozconfigs and your name comes up a bit under mobile/android/config/mozconfigs/*

could you have a look at this or suggest someone to review?

for context see: https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c6

Comment 38

[Jordan Lund (:jlund)]

I have been fighting regressions/errors in other projects, namely https://bugzil.la/1055918

Because of that, I was not able to get back to this today or yesterday. comment 36 seems promising, I'll allocate some time to this tomorrow. hopefully we can land https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c32 and start testing this out on cedar asap :)

Comment 39

[Jordan Lund (:jlund)]

reconfigured my staging master and pulled in latest m-c. triggered a bunch of tests to run over night. Will check in the morning.

/me crosses fingers

Comment 40

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #39)
> reconfigured my staging master and pulled in latest m-c. triggered a bunch
> of tests to run over night. Will check in the morning.
> 
> /me crosses fingers

crossing fingers worked. all the builds were green. I'm going to put this live on cedar once https://bugzilla.mozilla.org/attachment.cgi?id=8505259&action=edit is r+ and landed on m-c

Comment 41

[Jordan Lund (:jlund)]

Attachment: 20141028_1073772_split_apk_by_api.m-c.patch

last patch was inverted.

Comment 42

[Richard Newman [:rnewman]]

Comment on attachment 8512819 [details] [diff] [review]
20141028_1073772_split_apk_by_api.m-c.patch

Review of attachment 8512819 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me.

Comment 43

[Jordan Lund (:jlund)]

Comment on attachment 8512819 [details] [diff] [review]
20141028_1073772_split_apk_by_api.m-c.patch

https://hg.mozilla.org/integration/mozilla-inbound/rev/391254e1c290

Comment 44

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/391254e1c290

Comment 45

[Jordan Lund (:jlund)]

oops. you can tell how often I land things in tree.

Comment 46

[Jordan Lund (:jlund)]

Attachment: 20141030_1073772_split_apk_by_api-cedar-only.tools.patch

hmm, it looks like previous patches would have passed tests fine but those builders wouldn't have shown up on any masters. This hopefully fixes that.

kmoir, I think we want both api-9 and api-10 for all build + test linux masters (at least where we have 'android'). I noticed we have some masters that seem to be only for panda for tests? I only added api-10 for those ones since api-9 is our 2.3 constrained one.

Comment 47

[Jordan Lund (:jlund)]

Comment on attachment 8505257 [details] [diff] [review]
20141014_1073772_split_apk_by_api-cedar-only.bbotcfgs.patch

on default: http://hg.mozilla.org/build/buildbot-configs/rev/02f134ab655b

Comment 48

[Jordan Lund (:jlund)]

Comment on attachment 8514817 [details] [diff] [review]
20141030_1073772_split_apk_by_api-cedar-only.tools.patch

on default: http://hg.mozilla.org/build/tools/rev/9fd18e6f1388

Comment 49

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/02f134ab655b

Comment 50

[Richard Newman [:rnewman]]

Hey Jordan, what's next on the list to get this wrapped up?

Comment 51

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #50)
> Hey Jordan, what's next on the list to get this wrapped up?

verify builds + jobs in cedar

I triggered a round of jobs on Cedar last night. api-10 seemed to build fine and tests for the most part are green. I don't know the state of tests so we will want to verify the ones the failed are intermittent or failing regularly outside of api-10.

api-9 was a different story. It failed to compile: https://tbpl.mozilla.org/php/getParsedLog.php?id=51736200&full=1&branch=cedar

api-9 build snippet:
/builds/slave/ced-and-api-9-0000000000000000/build/mobile/android/base/resources/layout/new_tablet_tabs_item_cell.xml:24: error: Error: No resource found that matches the given name (at 'textColor' with value '@color/new_tablet_tab_item_title').
/builds/slave/ced-and-api-9-0000000000000000/build/mobile/android/base/resources/layout/new_tablet_tabs_item_cell.xml:34: error: Error: No resource found that matches the given name (at 'src' with value '@drawable/new_tablet_tab_item_close_button').


I merged the latest m-c into cedar so it appears like it could be a new regression?

mcomella, any ideas?

Comment 52

[Richard Newman [:rnewman]]

Should be fixed by Bug 1093394.

Comment 53

[Jordan Lund (:jlund)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1093394#c6 fixes bug 1083394.

triggered new round of android builds on cedar: https://tbpl.mozilla.org/?tree=Cedar&rev=b191b0861d8a

Comment 54

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #53)
> https://bugzilla.mozilla.org/show_bug.cgi?id=1093394#c6 fixes bug 1083394.
> 
> triggered new round of android builds on cedar:
> https://tbpl.mozilla.org/?tree=Cedar&rev=b191b0861d8a

failed with      [aapt] /builds/slave/ced-and-api-9-0000000000000000/build/obj-firefox/mobile/android/geckoview_library/res/layout-v11/new_tablet_tabs_item_cell.xml:25: error: Error: No resource found that matches the given name (at 'textColor' with value '@color/new_tablet_tab_item_title').

looks like bug 1091673 this time? odd though as I think cedar is up to date with m-c?

Comment 55

[Jordan Lund (:jlund)]

rnewman, looks like compile worked but something was missing for package target. https://bugzil.la/1091673 and https://bugzil.la/1093394 both touch new_tablet_tab_item_title

Comment 56

[Richard Newman [:rnewman]]

Bug 1093826.

The build produces a GeckoView library. That doesn't make sense in a constrained build, but there's still some wrangling over how best to turn it off.

(Sorry that this is whack-a-mole.)

Comment 57

[Richard Newman [:rnewman]]

jlund: looks like the answer is to add

export MOZ_DISABLE_GECKOVIEW=1

to the config for that build. That should get you moving.

Comment 58

[Jordan Lund (:jlund)]

Attachment: 141106_bug_1073772_split_apk_disable_gecko_view_api9.patch

I've respun just the api-9 builders on cedar: https://tbpl.mozilla.org/?tree=Cedar&rev=1c17d0767ad2

feel free to delay review until the results are in :)

Comment 59

[Richard Newman [:rnewman]]

Builds are working. I've retriggered a bunch of failing jobs, because emulators.

Comment 60

[Richard Newman [:rnewman]]

I think I might know what some of those failures are: we don't ship Clear Sans in these builds, so anything that renders text and does a per-pixel comparison, or anything that relies on a font's bounds, is going to fail.

Comment 61

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #60)
> I think I might know what some of those failures are: we don't ship Clear
> Sans in these builds, so anything that renders text and does a per-pixel
> comparison, or anything that relies on a font's bounds, is going to fail.

as per irc, I've triggered a build on cedar without disable-android-include-fonts for the api-9 based builds.

this won't be going into production but is instead for testing

https://tbpl.mozilla.org/?tree=Cedar&rev=122ca4bc5067

Comment 62

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Jordan Lund (:jlund) from comment #61)
> (In reply to Richard Newman [:rnewman] from comment #60)
> > I think I might know what some of those failures are: we don't ship Clear
> > Sans in these builds, so anything that renders text and does a per-pixel
> > comparison, or anything that relies on a font's bounds, is going to fail.
> 
> as per irc, I've triggered a build on cedar without
> disable-android-include-fonts for the api-9 based builds.
> 
> this won't be going into production but is instead for testing
> 
> https://tbpl.mozilla.org/?tree=Cedar&rev=122ca4bc5067

Looking very green

Comment 63

[Richard Newman [:rnewman]]

Yup, I think my hypothesis has been proven.

So let's move forward with this without the font flag, jlund, and I'll file a follow up to figure out how to make that happen.

Thanks for doing that testing!

Comment 64

[Jordan Lund (:jlund)]

(In reply to Richard Newman [:rnewman] from comment #63)
> Yup, I think my hypothesis has been proven.
> 
> So let's move forward with this without the font flag, jlund, and I'll file
> a follow up to figure out how to make that happen.
> 
> Thanks for doing that testing!

np. glad we discovered the cause.

status update: FYI - I am at AWS re:invent all week this week so work needed from my end: namely bug 1080749, will slow to a halt. I may have some time friday/sat to look into it

Comment 65

[Richard Newman [:rnewman]]

Thanks for keeping us posted!

Comment 66

[Jordan Lund (:jlund)]

k, had some progress with this today.

nightlies (mozharness script call step included) work with the exception of the update call. I am still trying to interpret the balrog bits specific to android. Should have that resolved by tomorrow assuming i can continue down this route: https://bugzilla.mozilla.org/show_bug.cgi?id=1080749#c7

one thing though I think is wrong, we are not running 4.0 debug tests on cedar. IIUC, we should be running 4.0 debug tests on release branches + cedar. At least that's how I interpret: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/mobile_config.py?rev=22019f9ea388#1773

Comment 67

[Jordan Lund (:jlund)]

Attachment: 141122_bug_1073772_split_apk_disable_gecko_view_and_fonts_api9.patch

I've set up ash for split-apk and nightly builds. I had to land the gecko view and font changes for api-9 on ash (this patch)

You have already r+'d the gecko_view part but I just want to confirm that we still want to go ahead with the font part landing on m-c and then you are going to follow up with it later: https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c63

if you r+ this, I'll just land the whole thing on m-c asap.

Comment 68

[Jordan Lund (:jlund)]

Comment on attachment 8527255 [details] [diff] [review]
141122_bug_1073772_split_apk_disable_gecko_view_and_fonts_api9.patch

mfinkle: just noticed rnewman is on pto. could you approve this or suggest someone. thanks!

Comment 69

[Richard Newman [:rnewman]]

Comment on attachment 8527255 [details] [diff] [review]
141122_bug_1073772_split_apk_disable_gecko_view_and_fonts_api9.patch

Review of attachment 8527255 [details] [diff] [review]:
-----------------------------------------------------------------

For this I'll click through the email!

Comment 70

[Jordan Lund (:jlund)]

Comment on attachment 8527255 [details] [diff] [review]
141122_bug_1073772_split_apk_disable_gecko_view_and_fonts_api9.patch

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c4134bbaa3f

Comment 71

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6c4134bbaa3f

Comment 72

[Jordan Lund (:jlund)]

Attachment: 141205_split_apk-rm_panda_ref_from_api_9-bbotcfgs.patch

r=kmoir

Comment 73

[Jordan Lund (:jlund)]

so a bit of an update,

this landed on trunk today via https://bugzil.la/1080749

 - reftest api-9 jobs were stuck pending. This[1] should address that
 - api-10 debug jobs are not being scheduled. currently working on a patch to fix that. aiming to resolve by tomorrow morning
 - fennec nightly updates are frozen so updates won't be served. This is on purpose so the next (tomorrow 3am) nightlies can be tested and verified that they will be served to the right device based on api. devices will be served tomorrow before noon, after that verification.

[1] https://bugzilla.mozilla.org/attachment.cgi?id=8532701

Comment 74

[Jordan Lund (:jlund)]

Attachment: 141205_split_apk-fix_api_10_debug.patch

Okay, this should fix issue where api 10+ debug jobs were not being triggered

the issue was that the 'branch' the api-10 debug build was sending as part of the sendchange wasn't matching the change filter of the associated tests


here is what buildbot was expecting:
    <buildbot.schedulers.basic.Scheduler> {'builderNames': ['Android 4.0 armv7 API 10+ fx-team debug test mochitest-1',
                  'Android 4.0 armv7 API 10+ fx-team debug test mochitest-2',
                  'Android 4.0 armv7 API 10+ fx-team debug test mochitest-3',
                  'Android 4.0 armv7 API 10+ fx-team debug test mochitest-4',
                  'Android 4.0 armv7 API 10+ fx-team debug test mochitest-5',
                  'Android 4.0 armv7 API 10+ fx-team debug test mochitest-6',
                  # ...
                  # ... etc
                  ],
 'change_filter': <ChangeFilter on branch == fx-team-android-api-10-debug-unittest>,
 'fileIsImportant': None,
 'name': 'tests-fx-team-panda_android-debug-unittest',
 'properties': {'scheduler': 'tests-fx-team-panda_android-debug-unittest'},
 'treeStableTimer': None}


however the sendchange was doing:

'buildbot', 'sendchange', '--master', 'buildbot-master81.build.mozilla.org:9301', '--username', 'sendchange-unittest', '--branch', 'fx-team-android-debug-api-10-unittest', # etc ...

tl;dr fx-team-android-debug-api-10-unittest (sendchange) != fx-team-android-api-10-debug-unittest (change_filter) which was bad.

Comment 75

[Jordan Lund (:jlund)]

Comment on attachment 8532761 [details] [diff] [review]
141205_split_apk-fix_api_10_debug.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/bbea57341f22

Comment 76

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/c2cbfc412a78

Comment 77

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/bbea57341f22

Comment 78

[Geoff Brown [:gbrown]]

We'll need to update trychooser and tell folks about the change to try build names: https://gbrownmozilla.wordpress.com/2014/12/07/new-android-job-names-on-treeherder-update-your-try-pushes/.

Comment 79

[Jordan Lund (:jlund)]

(In reply to Geoff Brown [:gbrown] from comment #78)
> We'll need to update trychooser and tell folks about the change to try build
> names:
> https://gbrownmozilla.wordpress.com/2014/12/07/new-android-job-names-on-
> treeherder-update-your-try-pushes/.

thanks, I did not catch that till after this landed. The trychooser fix is being tracked in one of the dep bugs to this: bug 1108058

Comment 80

[Richard Newman [:rnewman]]

In the mean time, I mailed mobile-firefox-dev, so that should catch many of the affected devs.

Comment 81

[Mark Finkle (:mfinkle) (use needinfo?)]

Sorry for asking a stupid question here, when we've almost got this wrapped up. We originally wanted to support two APKs: One for Gingerbread and one for everything newer.

Gingerbread = API 9 and API 10 [1]
Everything newer = API 11 and higher

Do we need to tweak the changes to handle API 10 as Gingerbread?

[1] http://en.wikipedia.org/wiki/Android_version_history

Comment 82

[Jordan Lund (:jlund)]

(In reply to Mark Finkle (:mfinkle) from comment #81)
> Sorry for asking a stupid question here, when we've almost got this wrapped
> up. We originally wanted to support two APKs: One for Gingerbread and one
> for everything newer.
> 
> Gingerbread = API 9 and API 10 [1]
> Everything newer = API 11 and higher
> 

I spoke with Mark this morning. We are going to split by 'all of gingerbread' now and not just api-9.

So instead of (from comment 6):
> 10+ (current ARMv7 config named "android"):
> 
> ac_add_options --with-android-min-sdk=10
> 
> 9 (clone of current ARMv7 "android", name it "android-9-constrained"):
> 
> ac_add_options --with-android-min-sdk=9
> ac_add_options --with-android-max-sdk=9

we will do:

11+ (current ARMv7 config named "android"):

ac_add_options --with-android-min-sdk=11

9-10 (clone of current ARMv7 "android", name it "android-9-10"):

ac_add_options --with-android-min-sdk=9
ac_add_options --with-android-max-sdk=10

While this is a simple mozconfig flip, all of the releng code and naming schemes will have to change.



the rollout plan:

1) test new mozconfigs once on a twig branch. I will land a quick patch that only changes the mozconfigs from the requested while I fix the naming schemes.

2) figure out new naming scheme for treeherder/releng-configs. ryanvm, wrt treeherder, in comment 24 you suggested naming. what would you like now that suggests api-9-10 and api-11+? fwiw, 9 + 10 makes up gingerbread so we could use 'gb' but that doesn't help the api-11+ name. treeherder/tbpl bugs 1078265 1078266 this affects.

3) if step 1 is successful, land patches for:
    a) tools(trychooser + balrog)
    b) mozharness(multi/single locale)
    c) buildbot-configs/mozilla-central

4) change balrog rules. originally it was https://bugzilla.mozilla.org/show_bug.cgi?id=1080749#c8 but now we need to add 2.3.3, 2.3.4, 2.3.5, 2.3.5, 2.3.6, 2.3.7 according to http://en.wikipedia.org/wiki/Android_version_history#Android_2.3.3.E2.80.932.3.7_Gingerbread_.28API_level_10.29

4) spin new nightlies

ni: ryanvm for question in step 2

Comment 83

[Ryan VanderMeulen [:RyanVM]]

Discussed on IRC a bit, and I think it'd be simplest to just switch the names from API9/API10+ to API10/API11+.

For a future scenario with more API cutoffs, we could use the max SDK version and a + for the "top" version. So something like API10, API15, API20, API21+ (or whatever, you get the idea). Does that seem sensible to those who understand the complexities of Android API versions way better than I do?

Comment 84

[Kevin Brosnan [Ex-Mozilla]]

The minimum API number is more important as the file will not install on lower versions. An API 8 phone can not install an API 9 APK.

Comment 85

[Ryan VanderMeulen [:RyanVM]]

So in that case, we'd go with API9 and API11+ in this situation. That seems sensible to me.

Comment 86

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #85)
> So in that case, we'd go with API9 and API11+ in this situation. That seems
> sensible to me.

+1

Comment 87

[Jordan Lund (:jlund)]

Attachment: 141208_split_apk-new_api_11_split-mozconfigs-mc.patch

first patch up

here is a try push reflecting such changes: https://tbpl.mozilla.org/?tree=Try&rev=6f713678da7e

Note that the buildernames don't express the changes (I can't change any of that without buidlbot/mozharness/tools changes). the api-9 is api-9-10 mozconfigs and the api-10 is api-11

Comment 88

[Jordan Lund (:jlund)]

Attachment: 141208_split_apk-new_api_11_split-bbotcfgs.patch

sorry kim, need to iterate again on this. see https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c82 for context

see https://bugzilla.mozilla.org/show_bug.cgi?id=1073772#c87 for mozconfig changes and subsequent try push

Comment 89

[Jordan Lund (:jlund)]

Attachment: 141208_split_apk-new_api_11_split-prod-masters_json-tools.patch

covers the api-10 -> api-11 changes plus fixes a mistake where I don't think api-10 should be included with the linux test masters

Comment 90

[Jordan Lund (:jlund)]

other api-11 patches landing in Bug 1080749 mozharness single locale + balrog tools fix) and bug 1108058 (trychooser tools fix)

Comment 91

[Richard Newman [:rnewman]]

(In reply to Mark Finkle (:mfinkle) from comment #81)
> Sorry for asking a stupid question here, when we've almost got this wrapped
> up. We originally wanted to support two APKs: One for Gingerbread and one
> for everything newer.

The original context for this wasn't directly to fix things like Bug 1100361; it was a first start at stripping things that weren't needed on various platforms.

API 9 is a good first target in that it had a lot of conditional code and there are no API 9 tablets, so we can exclude an even broader range of resources.

"Gingerbread" was an imprecise (or overly precise, versus "MR1") way of referring to API 9. Next on the line were 10-13 (phones and tablets).

If we're now in a situation where we need to fix API 10 issues in the same way, or we're more urgently targeting constrained devices, then we can spread the first build -- but we should be aware that the constrained build as currently specced might exclude things that are needed for API 10 tablets.

Comment 92

[Richard Newman [:rnewman]]

Comment on attachment 8533546 [details] [diff] [review]
141208_split_apk-new_api_11_split-mozconfigs-mc.patch

Review of attachment 8533546 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/config/mozconfigs/android-api-9-10-constrained/debug
@@ +9,1 @@
>  ac_add_options --enable-android-resource-constrained

This line (and its sibling in the nightly config) is the only thing I'm worried about.

We don't explicitly exclude tablet resources, but we do explicitly only include mdpi,hdpi as AAPT configurations, and I don't know if we've ever tested a constrained build on API 10.

That's almost certainly fine, but ideally I'd like this to be verified on an API 10 tablet if we have one.

Comment 93

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Richard Newman [:rnewman] from comment #92)

> That's almost certainly fine, but ideally I'd like this to be verified on an
> API 10 tablet if we have one.

Keep in mind that API 10 is 100% Gingerbread, not Honeycomb. As such, any tablet running API 10 would also have run API 9. I have an old one myself. One of the original Samsung Galaxy 7" Tabs. It runs GB like a phone.

Comment 94

[Jordan Lund (:jlund)]

Attachment: 141209_split_apk-new_api_11_split-mozconfigs-mc.patch

all patches are in place now. but just realized I can't land my last m-c patch as everything can't land together at the same time. we need to support the old mozconfigs through the transition.

Comment 95

[Richard Newman [:rnewman]]

Comment on attachment 8534570 [details] [diff] [review]
141209_split_apk-new_api_11_split-mozconfigs-mc.patch

Review of attachment 8534570 [details] [diff] [review]:
-----------------------------------------------------------------

Rubberstamp.

By the way, when this lands, I'm interested to know how much it costs to go from 9 only to 9-10.

Comment 96

[Jordan Lund (:jlund)]

Comment on attachment 8534570 [details] [diff] [review]
141209_split_apk-new_api_11_split-mozconfigs-mc.patch

https://hg.mozilla.org/integration/mozilla-inbound/rev/0311e3b1c374

Comment 97

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0311e3b1c374

Comment 98

[Andreas Pehrson [:pehrsons]]

Hi Jordan, I'm not sure if something is broken here or if I'm doing something wrong, but I am unable to run debug android builds on try. Opt seems to work fine. Debug seems to work fine on m-i and m-c as well.

"-b d -p android-api-10" should normally do the trick right?
See https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2ab4e16ef90e (where I also threw in some extra platforms after briefly scanning through this discussion and some of the patches, probably didn't know what I was doing..)

Comment 99

[Jordan Lund (:jlund)]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #98)
> Hi Jordan, I'm not sure if something is broken here or if I'm doing
> something wrong, but I am unable to run debug android builds on try. Opt
> seems to work fine. Debug seems to work fine on m-i and m-c as well.
> 
> "-b d -p android-api-10" should normally do the trick right?
> See https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2ab4e16ef90e
> (where I also threw in some extra platforms after briefly scanning through
> this discussion and some of the patches, probably didn't know what I was
> doing..)

hmm, I suspect what is happening is the 'd' option is telling the given platform 'android-api-10' to append '-debug' to it. but in our infra, I have the debug platform defined as 'android-debug-api-10'. let me change things from the buildbot end to reflect what trychooser expects. thanks for the ping! patch incoming

Comment 100

[Jordan Lund (:jlund)]

Attachment: 141212_splitapk_fix_debug_pf_names-bbotcfgs.patch

interdiff of https://bugzilla.mozilla.org/attachment.cgi?id=8533547&action=edit

Comment 101

[Jordan Lund (:jlund)]

Comment on attachment 8533547 [details] [diff] [review]
141208_split_apk-new_api_11_split-bbotcfgs.patch

default https://hg.mozilla.org/build/buildbot-configs/rev/89f655f09e67

Comment 102

[Jordan Lund (:jlund)]

Comment on attachment 8533548 [details] [diff] [review]
141208_split_apk-new_api_11_split-prod-masters_json-tools.patch

default: https://hg.mozilla.org/build/tools/rev/4903d937c90e

Comment 103

[Jordan Lund (:jlund)]

Comment on attachment 8535745 [details] [diff] [review]
141212_splitapk_fix_debug_pf_names-bbotcfgs.patch

thanks! on default https://hg.mozilla.org/build/buildbot-configs/rev/4a87c58db287

Comment 104

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/89f655f09e67

Comment 105

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/4a87c58db287

Comment 106

[Jordan Lund (:jlund)]

Attachment: support both api 10 and 11 in slave health

I think this should address the slave health not grepping the api 11+ buildername.

I escaped + as I think js treats that as a regex symbol for repeating patterns and not the literal char. not confident in js regex so correct me if I am wrong :)

Comment 107

[Jordan Lund (:jlund)]

Comment on attachment 8536118 [details] [diff] [review]
support both api 10 and 11 in slave health

on default: https://hg.mozilla.org/build/slave_health/rev/5dbca835ad7a

Comment 108

[Andreas Pehrson [:pehrsons]]

I now got a debug build for platform "android-api-11", thanks!

You really should push the trychooser update though, a lot of people depend on it.

Comment 109

[Jordan Lund (:jlund)]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #108)
> I now got a debug build for platform "android-api-11", thanks!
> 
> You really should push the trychooser update though, a lot of people depend
> on it.

thanks. and done. trychooser updated to support api-11 now too

Comment 110

[Jordan Lund (:jlund)]

Attachment: api 11 nonunfied should copy mozconfigs from api 11 not 10

turns out I had a mistake in the newest split mozconfigs for the nonunified jobs

Comment 111

[Richard Newman [:rnewman]]

Comment on attachment 8538880 [details] [diff] [review]
api 11 nonunfied should copy mozconfigs from api 11 not 10

Review of attachment 8538880 [details] [diff] [review]:
-----------------------------------------------------------------

Good catch!

Comment 112

[Jordan Lund (:jlund)]

Comment on attachment 8538880 [details] [diff] [review]
api 11 nonunfied should copy mozconfigs from api 11 not 10

https://hg.mozilla.org/integration/mozilla-inbound/rev/7a5db616bc57

Comment 113

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7a5db616bc57

Comment 114

[Jordan Lund (:jlund)]

remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/fec2872902f1
remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/fa1d2c3ef665

Comment 115

[Jordan Lund (:jlund)]

remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/810596766478
remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/d1d3584dade2

Comment 116

[Jordan Lund (:jlund)]

remote:   https://hg.mozilla.org/releases/mozilla-release/rev/a7d8aad2a696
remote:   https://hg.mozilla.org/releases/mozilla-release/rev/cbe57d2c6cbf

Comment 117

[Jordan Lund (:jlund)]

Attachment: 150102_releng_split_apk-fix_buildernames-bbotconfigs.patch

now that tbpl/treeherder split-apk bits are in production[1], we can fix the builder names to be more accurate.

caution note wrt deploying this patch. Due to pandas behaviour, this patch may require a mass reboot of pandas after a reconfig.

when this patch is deployed, we should close this bug and follow up with train riding bugs like bug 1117241

[1] current revs are new than my pushes: https://treeherder.mozilla.org/media/revision

Comment 118

[Jordan Lund (:jlund)]

Comment on attachment 8543459 [details] [diff] [review]
150102_releng_split_apk-fix_buildernames-bbotconfigs.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/b9c34177b915

Comment 119

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b9c34177b915

Comment 120

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.