1006082 - Run Android 2.3 tests against armv6 builds, on Ash only

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

I want to investigate running tests on the Android 2.3 emulator using Armv6 builds. I have completed a trial run on an ix loaner with good results. As a next step, I want to run them on Ash.

There are armv6 builds on mozilla-central currently ("Android 2.2 Armv6 Opt"), but it looks like they do not run on Ash. So running the Android 2.2 Armv6 Opt build on Ash is probably the first step here. Then we want to run all the test suites just the same way we do for Android 2.3 Opt, but using the Armv6 build.

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1006082puppet.patch

puppet patch to add platform to avoid duplicate buider name issues

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: bug1006082bb.patch

So this was the approach that I was using to add the Android 2.3 tests to run on emulators after the armv6 builds are invoked. The problem is that the Android 2.3 tests on emulators are defined via mozharness and armv6 platform is in the non-mozharness format.  So this patch doesn't work if tests are defined like this on line 1388 

 'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),

however it does add the correct builders if I do it like this
 'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),

which obviously isn't the right list of test suites, but is the right format.

Aki, I'm just wondering what you would suggest to do in this case to define the tests in a way that could be run on this platform

Comment 3

[Kim Moir [:kmoir] ET]

Aki here's the error message with the checkconfig

(test2)[[e-mail] test2]$ make checkconfig
cd master && /builds/buildbot/kmoir/test2/bin/buildbot checkconfig
/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
  import MimeWriter, tempfile, rfc822
Traceback (most recent call last):
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/master.py", line 652, in loadConfig
    exec f in localDict
  File "/builds/buildbot/kmoir/test2/master/master.cfg", line 173, in <module>
    MOBILE_BRANCH_UNITTEST_VARS['platforms'])
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbotcustom/misc.py", line 2663, in generateTalosBranchObjects
    for suites_name, suites in branch_config['platforms'][platform][slave_platform][unittest_suites]:
TypeError: list indices must be integers, not str
make: *** [checkconfig] Error 1

Comment 4

[Aki Sasaki (not active)]

(In reply to Kim Moir [:kmoir] from comment #2)
> Created attachment 8423923 [details] [diff] [review]
> bug1006082bb.patch
>
> So this was the approach that I was using to add the Android 2.3 tests to
> run on emulators after the armv6 builds are invoked. The problem is that the
> Android 2.3 tests on emulators are defined via mozharness and armv6 platform
> is in the non-mozharness format.  So this patch doesn't work if tests are
> defined like this on line 1388
>
>  'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>
> however it does add the correct builders if I do it like this
>  'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),
>
> which obviously isn't the right list of test suites, but is the right format.
>
> Aki, I'm just wondering what you would suggest to do in this case to define
> the tests in a way that could be run on this platform


I haven't tested to verify, but looks like ANDROID_NOWEBGL_UNITTEST_DICT looks like
{
    'opt_unittest_suites': [(..., {}), ...],
    'debug_unittest_suites': [(..., {}), ...],
}

and ANDROID_2_3_MOZHARNESS_DICT looks like

[(..., {}), ...]

So instead of

        'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),

try

        'ubuntu64_hw_armv6_mobile': {
            'opt_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
            'debug_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
        },

?

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1006082bb-3.patch

will attach builder diff

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1006082.diff

builder diff

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: bug1006082bb-3.patch

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 8429493 [details] [diff] [review]
bug1006082bb-3.patch

Review of attachment 8429493 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +88,4 @@
>  PLATFORMS['android-armv6']['env_name'] = 'android-perf'
>  PLATFORMS['android-armv6']['is_mobile'] = True
>  PLATFORMS['android-armv6']['tegra_android-armv6'] = {'name': "Android 2.2 Armv6 Tegra"}
> +PLATFORMS['android-armv6']['ubuntu64_hw_armv6_mobile'] = {'name': "Android 2.2 Emulator on ix"}

s/2.2/2.3/ ?

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug1006082bb-4.patch

regarding comment #11 it would have to be something like 

Android 2.3 Emulator on ix for armv6

if you just change it to 2.3 you'll have a duplicate builder error

It's confusing because we're running 2.3 tests against a 2.2 build

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 8429546 [details] [diff] [review]
bug1006082bb-4.patch

>         'tegra_android-armv6': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),
>+        'ubuntu64_hw_armv6_mobile': {
>+            'opt_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>+            'debug_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>+        },
<snip>
>+# bug 1006082 Run Android 2.3 tests against armv6 builds, on Ash only
>+for branch in BRANCHES:
>+    if branch in ('ash',):
>+        continue
>+    for platform in BRANCHES[branch]['platforms']:
>+        if not platform in PLATFORMS:
>+            continue
>+        if not platform == ('android-armv6'):
>+            continue
>+        if 'ubuntu64_hw_armv6_mobile' in BRANCHES[branch]['platforms']['android-armv6']: 
>+            del BRANCHES[branch]['platforms']['android-armv6']['ubuntu64_hw_armv6_mobile']

This patch adds all the builders to every branch, and then removes them from every branch but ash.
If we plan on adding more branches shortly, this may be the fastest path to victory, since we only have to extend the tuple to add those additional branches... otherwise we could just have empty test suites as the default and add the ANDROID_2_3_MOZHARNESS_DICT to the {opt,debug}_unittest_suites on ash.

... I find it kind of odd that we're not also targeting Cedar, but I imagine that will be a next step.

>+# otherwise spurious builders are created on ash
>+# part of bug 1006082 Run Android 2.3 tests against armv6 builds, on Ash only
>+del BRANCHES['ash']['platforms']['android-armv6']['tegra_android-armv6']

I think ash is used for general "mozharness-try", not reserved for armv6 emulator work, so do we really want to get rid of these?  If we're going to be done with this work in the next week or so I don't have a big problem with this.  If we're going to fork ash and have it be both a test greening area (which Cedar should be) and a mozharness-try area, we may run into difficulties.

r+ with caveats.

Comment 11

[Geoff Brown [:gbrown]]

I did most of the original Android 2.3 work on ash, since there were a lot of mozharness changes for that. I requested ash-only for armv6 to be consistent with that. Also, cedar has been busy lately with jgraham's work and I thought I would stay out of his way.

Based on my loaner experience, I don't expect to need to do much work at all for armv6 -- just a few builds to verify everything is fine, or perhaps a week of test greening work.

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug1006082bb-5.patch

I like the approach you suggested better where the default tests are blank and then we add them to ash.   The builder diff looks good and test-masters passes too.

Comment 13

[Chris Cooper [:coop] (he/him)]

Merged to production and deployed.

Comment 14

[Kim Moir [:kmoir] ET]

Attachment: bug1006082fix.patch

The previous patch works in production 

It looks good on tbpl except the armv6 test jobs have been consolidated with the armv7 jobs under "Android 2.3 Opt"

I think this fixes that issue.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 8431669 [details] [diff] [review]
bug1006082fix.patch

Haha.

Comment 16

[Kim Moir [:kmoir] ET]

This was deployed earlier this morning in a reconfig, looks good.

Comment 17

[Kim Moir [:kmoir] ET]

The label looks good but the test jobs all show as pending.  They have been pending for 4+ hours, investigating.

Comment 18

[Kim Moir [:kmoir] ET]

Attachment: bug1006082tools.patch

The problem was that there wasn't a test master with both armv6 and linux64 enabled thus no builders were created nor slaves connected.  I didn't see this on my test master because I was testing multiple bugs on it at the time and did have both of these configured in the json file for my master.  This didn't appear as a problem in my earlier bug where I enabled ix slaves for android builds because there are a masters with both the android platform and linux64 slaves, but not android-armv6.

Comment 19

[Kim Moir [:kmoir] ET]

verified in tbpl

Comment 20

[Simone Bruno [:simone]]

This went live with reconfig on 2014-06-02 07:03 PT