867377 - Support crashtests and jsreftests for Android x86 on Cedar

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Dan Minor [:dminor]]

Crashtests and jsreftests seem to run stably on Android x86 in emulation and would be a good candidate to start setting up a mozharness target on ubuntu ix. Reftests also run but have a few unexpected results that need to be sorted out.

I have a emulator image with sutagent and watcher working which I can make available. I also have a simple script to start up one or more emulators and remap the ports so that multiple emulators can run on a single host.

When running the tests, I'm using these arguments in addition to the ones normally used for Android, which may or may not be similar to b2g:
--http-port=8890 --ssl-port=4444 (I change these to avoid conflicts when running multiple emulators on a single host)
--devicePort=20701 (I set up a different redirect port for each emulator)
--remote-webserver=10.0.2.2 (ip address of localhost when connecting from emulator)

Comment 1

[Dan Minor [:dminor]]

Attachment: Android x86 unit test script

Attached is a first cut of a unit test script based upon the B2G work. Any feedback would be appreciated.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 749859 [details]
Android x86 unit test script

Cool, thanks!
My main question would be, does it work?

>from mozharness.mozilla.testing.device import ADBDeviceHandler

Nit: trailing whitespace; I found 4 instances of this -- lines 25, 110, 175, 176

>        [["--xre-url"],
>        {"action": "store",
>         "dest": "xre_url",
>         "default": None,
>         "help": "URL to the desktop xre zip",
>        }],

Do you know if we can use the same xre as the b2g emulator tests?

>        [["--total-chunks"],
>        {"action": "store",
>         "dest": "total_chunks",
>         "help": "Number of total chunks",
>        }],
>        [["--this-chunk"],
>        {"action": "store",
>         "dest": "this_chunk",
>         "help": "Number of this chunk",
>        }]] + copy.deepcopy(testing_config_options)
>
>    error_list = [
>        {'substr': 'FAILED (errors=', 'level': ERROR},
>        {'substr': r'''Could not successfully complete transport of message to Gecko, socket closed''', 'level': ERROR},
>        {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
>        {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
>    ]
>
>    virtualenv_requirements = [
>    ]
>
>    virtualenv_modules = [
>    ]

You don't need marionette or anything?
If the virtualenv modules and requirements are, in fact, zero, I'm wondering if we even need VirtualenvMixin.
(It might be that a parent mixin requires it; not sure.)

>        # these are necessary since self.config is read only
>        c = self.config
>        self.adb_path = c.get('adb_path', self._query_adb())

You could set the default for --adb-path to be self.which('adb') or os.getenv('ADB_PATH'), and be able to still override it via config file or command line.
e.g.

        [["--adb-path"],
        {"action": "store",
         "dest": "adb_path",
         "default": self.which('adb') or os.getenv('ADB_PATH'),
         "help": "Path to adb",
        }],

>        self.busybox_path = None

Doesn't look like you use busybox_path?

>    def install(self):
>        dh = ADBDeviceHandler(config={'device-id': self.config['device_id'], 'enable_automation': True, 'device_package_name': self.config['application']})
>        dh.device_id = self.config['device_id']
>
>        #attempt to remove fennec and reinstall
>        dh.uninstall_app(self.config['application'])
>        dh.install_app(self.installer_path)

Is this the only place you plan on using the ADBDeviceHandler?
If you want to re-use it, you probably want to save the handler as self.dh or something.


Do you need a config file?

Comment 3

[Dan Minor [:dminor]]

(In reply to Aki Sasaki [:aki] from comment #2)
> Comment on attachment 749859 [details]
> Android x86 unit test script
> 
> Cool, thanks!
> My main question would be, does it work?

It works fine on my laptop. I'm hoping to get a loaner ubuntu ix machine to do some testing in an environment that more closely resembles production.

> 
> >from mozharness.mozilla.testing.device import ADBDeviceHandler
> 
> Nit: trailing whitespace; I found 4 instances of this -- lines 25, 110, 175,
> 176
> 

Fixed.

> >        [["--xre-url"],
> >        {"action": "store",
> >         "dest": "xre_url",
> >         "default": None,
> >         "help": "URL to the desktop xre zip",
> >        }],
> 
> Do you know if we can use the same xre as the b2g emulator tests?
> 

Yes, I'm using the same xre.

> >        [["--total-chunks"],
> >        {"action": "store",
> >         "dest": "total_chunks",
> >         "help": "Number of total chunks",
> >        }],
> >        [["--this-chunk"],
> >        {"action": "store",
> >         "dest": "this_chunk",
> >         "help": "Number of this chunk",
> >        }]] + copy.deepcopy(testing_config_options)
> >
> >    error_list = [
> >        {'substr': 'FAILED (errors=', 'level': ERROR},
> >        {'substr': r'''Could not successfully complete transport of message to Gecko, socket closed''', 'level': ERROR},
> >        {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
> >        {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
> >    ]
> >
> >    virtualenv_requirements = [
> >    ]
> >
> >    virtualenv_modules = [
> >    ]
> 
> You don't need marionette or anything?
> If the virtualenv modules and requirements are, in fact, zero, I'm wondering
> if we even need VirtualenvMixin.
> (It might be that a parent mixin requires it; not sure.)
> 

It appears I do need to create a virtualenv (the tests fail without it.)

> >        # these are necessary since self.config is read only
> >        c = self.config
> >        self.adb_path = c.get('adb_path', self._query_adb())
> 
> You could set the default for --adb-path to be self.which('adb') or
> os.getenv('ADB_PATH'), and be able to still override it via config file or
> command line.
> e.g.
> 
>         [["--adb-path"],
>         {"action": "store",
>          "dest": "adb_path",
>          "default": self.which('adb') or os.getenv('ADB_PATH'),
>          "help": "Path to adb",
>         }],

I don't have a 'self' at this point, but there may be a nicer way of handling this. I'll take a look.

> 
> >        self.busybox_path = None
> 
> Doesn't look like you use busybox_path?
> 
No, that was left over from B2G, I've removed it.

> >    def install(self):
> >        dh = ADBDeviceHandler(config={'device-id': self.config['device_id'], 'enable_automation': True, 'device_package_name': self.config['application']})
> >        dh.device_id = self.config['device_id']
> >
> >        #attempt to remove fennec and reinstall
> >        dh.uninstall_app(self.config['application'])
> >        dh.install_app(self.installer_path)
> 
> Is this the only place you plan on using the ADBDeviceHandler?
> If you want to re-use it, you probably want to save the handler as self.dh
> or something.
> 
As of now, this is the only place I need this. I'll move it up to self if I end up using it in another place.

> 
> Do you need a config file?

Yes, but I'm still adding things to it, so I didn't post it for review.

Thanks for your comments!

Comment 4

[Dan Minor [:dminor]]

Attachment: Android x86 unit test script

Addressed earlier comments and added support for robocop tests.

Comment 5

[Dan Minor [:dminor]]

Attachment: Sample config file for Android x86 unit tests

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 753571 [details] [diff] [review]
Android x86 unit test script

>     def _build_mochitest_args(self):
>-        pass
>+        c = self.config
>+        dirs = self.query_abs_dirs()
>+        python = self.query_python_path('python')
>+        cmd = [
>+            python, os.path.join(dirs['abs_mochitest_dir'], 'runtestsremote.py'),
>+            '--autorun',
>+            '--close-when-done',
>+            '--dm_trans=sut',
>+            '--console-level', 'INFO',
>+            '--app', c['application'],
>+            '--remote-webserver', c['remote_webserver'],
>+            '--run-only-tests', self.test_manifest,
>+            '--xre-path', os.path.join(dirs['abs_xre_dir'], 'bin'),
>+            '--deviceIP', self.config['device_ip'],
>+            '--devicePort', self.config['device_port'],
>+            '--http-port', self.config['http_port'],
>+            '--ssl-port', self.config['ssl_port']
>+        ]
>+        cmd.extend(self._build_arg('--total-chunks', c.get('total_chunks')))
>+        cmd.extend(self._build_arg('--this-chunk', c.get('this_chunk')))
>+        cmd.extend(self._build_arg('--symbols-path', 'crashreporter-symbols.zip'))

Just FYI:
the _build_*_args() methods are semi-obsoleted by bug 872164.
These should work for now, but we may want to change behavior here to follow the same convention as bug 872164 later.
I'm fine having this land as-is for now, though.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 753571 [details] [diff] [review]
Android x86 unit test script

https://hg.mozilla.org/build/mozharness/rev/7caf49ab4dbe

Comment 8

[Aki Sasaki (not active)]

Landed https://hg.mozilla.org/build/mozharness/rev/5ef622ddcabe to silence pyflakes.

Comment 10

[Dan Minor [:dminor]]

Closing this out as bug 895186 is continuing this work.