Closed Bug 1006082 Opened 8 years ago Closed 7 years ago

Run Android 2.3 tests against armv6 builds, on Ash only

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: kmoir)

References

Details

Attachments

(5 files, 4 obsolete files)

I want to investigate running tests on the Android 2.3 emulator using Armv6 builds. I have completed a trial run on an ix loaner with good results. As a next step, I want to run them on Ash.

There are armv6 builds on mozilla-central currently ("Android 2.2 Armv6 Opt"), but it looks like they do not run on Ash. So running the Android 2.2 Armv6 Opt build on Ash is probably the first step here. Then we want to run all the test suites just the same way we do for Android 2.3 Opt, but using the Armv6 build.
Assignee: nobody → kmoir
Attached file bug1006082puppet.patch
puppet patch to add platform to avoid duplicate buider name issues
Attached patch bug1006082bb.patch (obsolete) — Splinter Review
So this was the approach that I was using to add the Android 2.3 tests to run on emulators after the armv6 builds are invoked. The problem is that the Android 2.3 tests on emulators are defined via mozharness and armv6 platform is in the non-mozharness format.  So this patch doesn't work if tests are defined like this on line 1388 

 'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),

however it does add the correct builders if I do it like this
 'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),

which obviously isn't the right list of test suites, but is the right format.

Aki, I'm just wondering what you would suggest to do in this case to define the tests in a way that could be run on this platform
Attachment #8423923 - Flags: feedback?(aki)
Aki here's the error message with the checkconfig

(test2)[kmoir@dev-master1.srv.releng.scl3.mozilla.com test2]$ make checkconfig
cd master && /builds/buildbot/kmoir/test2/bin/buildbot checkconfig
/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
  import MimeWriter, tempfile, rfc822
Traceback (most recent call last):
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbot-0.8.2_hg_4d9f780d62ff_production_0.8-py2.6.egg/buildbot/master.py", line 652, in loadConfig
    exec f in localDict
  File "/builds/buildbot/kmoir/test2/master/master.cfg", line 173, in <module>
    MOBILE_BRANCH_UNITTEST_VARS['platforms'])
  File "/builds/buildbot/kmoir/test2/lib/python2.6/site-packages/buildbotcustom/misc.py", line 2663, in generateTalosBranchObjects
    for suites_name, suites in branch_config['platforms'][platform][slave_platform][unittest_suites]:
TypeError: list indices must be integers, not str
make: *** [checkconfig] Error 1
(In reply to Kim Moir [:kmoir] from comment #2)
> Created attachment 8423923 [details] [diff] [review]
> bug1006082bb.patch
>
> So this was the approach that I was using to add the Android 2.3 tests to
> run on emulators after the armv6 builds are invoked. The problem is that the
> Android 2.3 tests on emulators are defined via mozharness and armv6 platform
> is in the non-mozharness format.  So this patch doesn't work if tests are
> defined like this on line 1388
>
>  'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>
> however it does add the correct builders if I do it like this
>  'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),
>
> which obviously isn't the right list of test suites, but is the right format.
>
> Aki, I'm just wondering what you would suggest to do in this case to define
> the tests in a way that could be run on this platform


I haven't tested to verify, but looks like ANDROID_NOWEBGL_UNITTEST_DICT looks like
{
    'opt_unittest_suites': [(..., {}), ...],
    'debug_unittest_suites': [(..., {}), ...],
}

and ANDROID_2_3_MOZHARNESS_DICT looks like

[(..., {}), ...]

So instead of

        'ubuntu64_hw_armv6_mobile': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),

try

        'ubuntu64_hw_armv6_mobile': {
            'opt_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
            'debug_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
        },

?
Attachment #8423923 - Flags: feedback?(aki)
Attached patch bug1006082bb-3.patch (obsolete) — Splinter Review
will attach builder diff
Attachment #8423923 - Attachment is obsolete: true
Attached file bug1006082.diff
builder diff
Attached patch bug1006082bb-3.patch (obsolete) — Splinter Review
Attachment #8429485 - Attachment is obsolete: true
Attachment #8423914 - Flags: review?(aki)
Attachment #8429493 - Flags: review?(aki)
Comment on attachment 8429493 [details] [diff] [review]
bug1006082bb-3.patch

Review of attachment 8429493 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +88,4 @@
>  PLATFORMS['android-armv6']['env_name'] = 'android-perf'
>  PLATFORMS['android-armv6']['is_mobile'] = True
>  PLATFORMS['android-armv6']['tegra_android-armv6'] = {'name': "Android 2.2 Armv6 Tegra"}
> +PLATFORMS['android-armv6']['ubuntu64_hw_armv6_mobile'] = {'name': "Android 2.2 Emulator on ix"}

s/2.2/2.3/ ?
Attachment #8423914 - Flags: review?(aki) → review+
Attached patch bug1006082bb-4.patch (obsolete) — Splinter Review
regarding comment #11 it would have to be something like 

Android 2.3 Emulator on ix for armv6

if you just change it to 2.3 you'll have a duplicate builder error

It's confusing because we're running 2.3 tests against a 2.2 build
Attachment #8429493 - Attachment is obsolete: true
Attachment #8429493 - Flags: review?(aki)
Attachment #8429546 - Flags: review?(aki)
Comment on attachment 8429546 [details] [diff] [review]
bug1006082bb-4.patch

>         'tegra_android-armv6': deepcopy(ANDROID_NOWEBGL_UNITTEST_DICT),
>+        'ubuntu64_hw_armv6_mobile': {
>+            'opt_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>+            'debug_unittest_suites': deepcopy(ANDROID_2_3_MOZHARNESS_DICT),
>+        },
<snip>
>+# bug 1006082 Run Android 2.3 tests against armv6 builds, on Ash only
>+for branch in BRANCHES:
>+    if branch in ('ash',):
>+        continue
>+    for platform in BRANCHES[branch]['platforms']:
>+        if not platform in PLATFORMS:
>+            continue
>+        if not platform == ('android-armv6'):
>+            continue
>+        if 'ubuntu64_hw_armv6_mobile' in BRANCHES[branch]['platforms']['android-armv6']: 
>+            del BRANCHES[branch]['platforms']['android-armv6']['ubuntu64_hw_armv6_mobile']

This patch adds all the builders to every branch, and then removes them from every branch but ash.
If we plan on adding more branches shortly, this may be the fastest path to victory, since we only have to extend the tuple to add those additional branches... otherwise we could just have empty test suites as the default and add the ANDROID_2_3_MOZHARNESS_DICT to the {opt,debug}_unittest_suites on ash.

... I find it kind of odd that we're not also targeting Cedar, but I imagine that will be a next step.

>+# otherwise spurious builders are created on ash
>+# part of bug 1006082 Run Android 2.3 tests against armv6 builds, on Ash only
>+del BRANCHES['ash']['platforms']['android-armv6']['tegra_android-armv6']

I think ash is used for general "mozharness-try", not reserved for armv6 emulator work, so do we really want to get rid of these?  If we're going to be done with this work in the next week or so I don't have a big problem with this.  If we're going to fork ash and have it be both a test greening area (which Cedar should be) and a mozharness-try area, we may run into difficulties.

r+ with caveats.
Attachment #8429546 - Flags: review?(aki) → review+
I did most of the original Android 2.3 work on ash, since there were a lot of mozharness changes for that. I requested ash-only for armv6 to be consistent with that. Also, cedar has been busy lately with jgraham's work and I thought I would stay out of his way.

Based on my loaner experience, I don't expect to need to do much work at all for armv6 -- just a few builds to verify everything is fine, or perhaps a week of test greening work.
I like the approach you suggested better where the default tests are blank and then we add them to ash.   The builder diff looks good and test-masters passes too.
Attachment #8429546 - Attachment is obsolete: true
Attachment #8429700 - Flags: review?(aki)
Attachment #8429700 - Flags: review?(aki) → review+
Attachment #8423914 - Flags: checked-in+
Attachment #8429700 - Flags: checked-in+
Merged to production and deployed.
The previous patch works in production 

It looks good on tbpl except the armv6 test jobs have been consolidated with the armv7 jobs under "Android 2.3 Opt"

I think this fixes that issue.
Attachment #8431669 - Flags: review?(aki)
Comment on attachment 8431669 [details] [diff] [review]
bug1006082fix.patch

Haha.
Attachment #8431669 - Flags: review?(aki) → review+
Attachment #8431669 - Flags: checked-in+
This was deployed earlier this morning in a reconfig, looks good.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
The label looks good but the test jobs all show as pending.  They have been pending for 4+ hours, investigating.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
The problem was that there wasn't a test master with both armv6 and linux64 enabled thus no builders were created nor slaves connected.  I didn't see this on my test master because I was testing multiple bugs on it at the time and did have both of these configured in the json file for my master.  This didn't appear as a problem in my earlier bug where I enabled ix slaves for android builds because there are a masters with both the android platform and linux64 slaves, but not android-armv6.
Attachment #8432949 - Flags: review?(aki)
Attachment #8432949 - Flags: review?(aki) → review+
Attachment #8432949 - Flags: checked-in+
verified in tbpl
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
This went live with reconfig on 2014-06-02 07:03 PT
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.