881293 - Run xpcshell tests on Android 4.0 try only, hidden

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

We want xpcshell tests to run on Android 4.0. They do not currently pass. It would be helpful to me if I could run xpcshell/Android 4.0 on try. Probably best if they are hidden for now.

Comment 1

[Justin Wood (:Callek)]

Attachment: [config] add it to try-only

This has been run through dump-master, and *looks* like it's correct for the buildbot side.

However it looks like we need to add stuff to http://mxr.mozilla.org/build/source/mozharness/configs/android/android_panda_releng.py and I'd love some assistance/guidance for what to change there, to support passing stuff like we do at http://mxr.mozilla.org/build/source/buildbotcustom/steps/unittest.py#720

Specifically it adds one job:

Android 4.0 Panda try opt test xpcshell ScriptFactory
    SetBuildProperty {'property_name': 'master', 'value': <function <lambda>>} {}
    C {'command': ['bash', '-c', 'pwd'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_basedir', 'property': 'basedir', 'strip': True, 'usePTY': 'slave-config', 'workdir': '.'} {}
    JSONPropertiesDownload {'blocksize': 16384, 'maxsize': None, 'mode': None, 'name': 'download_props', 's': None, 'slavedest': 'buildprops.json', 'workdir': '.'} {}
    C {'command': ['rm', '-rf', 'properties'], 'description': None, 'descriptionDone': None, 'log_eval_func': None, 'logfiles': {}, 'name': 'clobber_properties', 'usePTY': 'slave-config', 'workdir': '.'} {}
    C {'command': ['rm', '-rf', 'scripts'], 'description': None, 'descriptionDone': None, 'haltOnFailure': True, 'log_eval_func': <function eval_func>, 'logfiles': {}, 'name': 'clobber_scripts', 'usePTY': 'slave-config', 'workdir': '.'} {}
    MercurialCloneCommand {'command': ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'scripts'], 'description': None, 'descriptionDone': None, 'haltOnFailure': True, 'log_eval_func': <function eval_func>, 'logfiles': {}, 'name': 'clone_scripts', 'retry': False, 'timeout': 1260, 'usePTY': 'slave-config', 'workdir': '.'} {}
    C {'command': ['hg',  'update',  '-C',  '-r',  <WithProperties "%(script_repo_revision:-default)s">], 'description': None, 'descriptionDone': None, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'name': 'update_scripts', 'usePTY': 'slave-config', 'workdir': 'scripts'} {}
    C {'command': ['hg', 'id', '-i'], 'description': None, 'descriptionDone': None, 'extract_fn': None, 'haltOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'get_script_repo_revision', 'property': 'script_repo_revision', 'strip': True, 'usePTY': 'slave-config', 'workdir': 'scripts'} {}
    OutputStep {'data': <WithProperties "TinderboxPrint: mozharness_revlink: http://hg.mozilla.org/build/mozharness/rev/%(script_repo_revision)s">, 'log': 'output', 'name': 'tinderboxprint_script_revlink'} {}
    MockCommand {'command': ['/tools/buildbot/bin/python',  'scripts/scripts/android_panda.py',  '--cfg',  'android/android_panda_releng.py',  '--xpcshell-suite',  'xpcshell',  '--download-symbols',  'ondemand'], 'description': None, 'descriptionDone': None, 'env': {'MOZ_HIDE_RESULTS_TABLE': '1',  'PROPERTIES_FILE': <WithProperties "%(basedir)s/buildprops.json">}, 'haltOnFailure': True, 'log_eval_func': <function eval_func>, 'logfiles': {}, 'maxTime': 7200, 'mock': False, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': '%(basedir)s/', 'name': 'run_script', 'target': None, 'timeout': 3600, 'usePTY': 'slave-config', 'warnOnWarnings': True, 'workdir': '.'} {}
    C {'alwaysRun': True, 'command': ['bash', '-c', 'for file in `ls -1`; do cat $file; done'], 'description': None, 'descriptionDone': None, 'extract_fn': <function extractProperties>, 'flunkOnFailure': False, 'log_eval_func': None, 'logfiles': {}, 'name': 'set_script_properties', 'property': None, 'strip': True, 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': 'properties'} {}
    DisconnectStep {'alwaysRun': True, 'command': ['/tools/buildbot/bin/python',  'scripts/external_tools/count_and_reboot.py',  '-f',  '../reboot_count.txt',  '-n',  '1',  '-z'], 'description': 'reboot', 'descriptionDone': None, 'env': {'MOZ_HIDE_RESULTS_TABLE': '1',  'PROPERTIES_FILE': <WithProperties "%(basedir)s/buildprops.json">}, 'flunkOnFailure': False, 'force_disconnect': <function do_disconnect>, 'log_eval_func': None, 'logfiles': {}, 'name': 'reboot', 'usePTY': 'slave-config', 'warnOnFailure': False, 'workdir': '.'} {}

Comment 2

[Justin Wood (:Callek)]

Attachment: [mozharness] xpcshell config

So, this is a rough-mock up of what I suspect is right for a mozharness version of http://mxr.mozilla.org/build/source/buildbotcustom/steps/unittest.py#720

I'm happy if one of you guide me to a solution or take it and finish it up. This was not even close to tested, and :gbrown does say that android panda xpcshell is still broken, but needs it on try to get it fixed, so we need to have it run there first.

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 769355 [details] [diff] [review]
[config] add it to try-only

I forgot to mention that you need to add something to 
ANDROID_MOZHARNESS_XPCSHELL for the timeout and script_maxtime to reflect the correct values for that test suite. So r+ with that change.

Comment 4

[Kim Moir [:kmoir] ET]

Comment on attachment 769358 [details] [diff] [review]
[mozharness] xpcshell config

Looking at the tbpl log for xpcshell tests as a reference (I assume the parameters are the same) I think the config needs

--local-lib-dir ../fennec

for --apk you could use the value set in the mozharness script _query_abs_base_cmd for self.filename_apk 

I think you also need to add  "--symbols-path=%(symbols_path)s" to the config

The only way to really tests this is to run the tests on the dev-master and work out any issues.  If you want to test it on mine, let me know when you want to test it and I can back out my changes so it's clean for you to test on it.

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 769358 [details] [diff] [review]
[mozharness] xpcshell config

This eyeballs ok. I would agree with Kim that running it and comparing against an equivalent buildbot-driven xpcshell would probably be easiest to get things right.

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I managed to get xpcshell tests running with mozharness for the android x86 emulator: see https://bugzilla.mozilla.org/show_bug.cgi?id=895186#c15.

22:50:08     INFO - Running command: ['/home/cltbld/tests/scripts/scripts/build/venv/bin/python', 'remotexpcshelltests.py', '--deviceIP', '127.0.0.1', '--xre-path', '/home/cltbld/tests/scripts/scripts/build/hostutils', '--manifest', 'tests/xpcshell_android.ini', '--testing-modules-dir', '/home/cltbld/tests/scripts/scripts/build/tests/modules', '--apk', '/home/cltbld/tests/script     s/scripts/build/fennec-25.0a1.multi.android-i386.apk', '--no-logfiles'] in /home/cltbld/tests/scripts/scripts/build/tests/xpcshell
 22:50:08     INFO - Copy/paste: /home/cltbld/tests/scripts/scripts/build/venv/bin/python remotexpcshelltests.py --deviceIP 127.0.0.1 --xre-path /home/cltbld/tests/scripts/scripts/build/hostutils --manifest tests/xpcshell_android.ini --testing-modules-dir /home/cltbld/tests/scripts/scripts/build/tests/modules --apk /home/cltbld/tests/scripts/scripts/build/fennec-25.0a1.multi.android-i386.apk --no-logfiles
22:50:18     INFO -  Pushing assets/libmozalloc.so..
22:50:20     INFO -  Pushing assets/libnss3.so..
22:50:23     INFO -  Pushing assets/libxul.so..
22:50:42     INFO -  Pushing assets/libnssckbi.so..
22:50:42     INFO -  Pushing assets/libfreebl3.so..
22:50:42     INFO -  Pushing assets/libsoftokn3.so..
22:50:42     INFO -  Pushing assets/libomxplugin.so..
22:50:42     INFO -  Pushing assets/libomxplugingb.so..
22:50:43     INFO -  Pushing assets/libomxplugingb235.so..
22:50:43     INFO -  Pushing assets/libomxpluginhc.so..
22:50:43     INFO -  Pushing assets/libomxpluginsony.so..
22:50:43     INFO -  Pushing assets/libomxpluginfroyo.so..
22:50:43     INFO -  Pushing assets/libomxpluginjb-htc.so..
22:50:43     INFO -  Pushing lib/x86/libplugin-container.so..
22:50:43     INFO -  Pushing lib/x86/libmozglue.so..
22:53:07     INFO -  pushing /home/cltbld/tests/scripts/scripts/build/tests/xpcshell/tests
22:53:07     INFO -  TEST-INFO | /home/cltbld/tests/scripts/scripts/build/tests/xpcshell/tests/chrome/test/unit/test_abi.js | running test ...
22:53:09     INFO -  TEST-PASS | /home/cltbld/tests/scripts/scripts/build/tests/xpcshell/tests/chrome/test/unit/test_abi.js | test passed (time: 1946.577ms)

Hope that helps...I'm still interested in seeing these running on Panda.

Comment 7

[Kim Moir [:kmoir] ET]

gbrown,

I have patches to run them on the pandas but am running into an issue creating /data/local/xpcb (Bug 817235 describes some of the issues with this).  

0:33:05     INFO -    File "/builds/panda-0310/test/build/tests/xpcshell/devicemanager.py", line 375, in shellCheckOutput
10:33:05     INFO -      raise DMError("Non-zero return code for command: %s (output: '%s', retval: '%s')" % (cmd, output, retval))
10:33:05     INFO -  devicemanager.DMError: Non-zero return code for command: ['mkdir', '/data/local/xpcb'] (output: 'mkdir failed for /data/local/xpcb, Permission denied', retval: '255')
10:33:05    ERROR - Return code: 1

Is this a problem because we are running the tests on pandas instead of tegras?

http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20mozilla-central%20opt%20test%20xpcshell/builds/44/steps/run_script/logs/stdio

(using mozilla central branch instead of try builds for testing)

Comment 8

[Geoff Brown [:gbrown] (pto Apr 11-18)]

kmoir,

Thanks for pushing this forward! I think /data/local has different permissions and/or ownership on tegras and pandas. There is also a similar problem on the x86 emulator. In bug 817235, I should have made that shellCheckOutput use root. I'll try a simple patch this afternoon.

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug881293bbconfigs.patch

buildbot-configs tested so far

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug881293mozharness.patch

mozharness tested so far

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: xpcshellresults.txt

Thanks gbrown for the fix, I was able to get them running after your change in bug 902645.  I've attached the test results from this run, there are a few failures.  Do you want me to land the changes to enable them on try or would you like to address the test failures first?

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug881293bbconfigs-2.patch

Comment 13

[Geoff Brown [:gbrown] (pto Apr 11-18)]

That's great.  Please land your changes now. I will file bugs to follow-up on the failures.

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 788942 [details] [diff] [review]
bug881293bbconfigs-2.patch

Review of attachment 788942 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +890,5 @@
> +# Panda XPCShell on try only
> +for branch in BRANCHES:
> +    # Loop removes it from any branch that gets beyond here
> +    if branch in ('try', ):
> +        continue

just a little anecdote, I love how useful these loops I created end up being, I hate how verbose they are and how much we use them. I wish we never had to!

Comment 15

[Armen [:armenzg]]

Code live: merged to production.

Comment 16

[Armen [:armenzg]]

Mass back out from production.
We should figure out in Cedar if this is involved with the tree burning.

Comment 17

[Armen [:armenzg]]

Merged again to the production branch of mozharness.
Live as of ~7:40AM PDT.

Comment 18

[Kim Moir [:kmoir] ET]

verified in tbpl

Comment 19

[Ed Morley [:emorley]]

Were showing on Try; hidden.