749011 - Add support for running mochitest-plain on B2G

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Jonathan Griffin (:jgriffin)]

We want to be able to run mochitest-plain on B2G.  To do that, we'll need B2G versions of remoteautomation.py and runtestsremote.py, among other things.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we don't run ssltunnel tests remotely or websocket tests remotely.  Currently we don't run mochitest-chrome and for browser-chrome everything is bundled in an extension and run off the device only.

Also to consider is the host machine running the tests and serving the files via http.  

If our core changes to the system can happen in devicemanager*.py, then we don't have to change much in our harnesses.  For Robocop I had to change harnesses and devicemanager*.py, I expect similar stuff here as well.

Comment 2

[Jonathan Griffin (:jgriffin)]

(In reply to Joel Maher (:jmaher) from comment #1)
> we don't run ssltunnel tests remotely or websocket tests remotely. 
> Currently we don't run mochitest-chrome and for browser-chrome everything is
> bundled in an extension and run off the device only.
> 
> Also to consider is the host machine running the tests and serving the files
> via http.  
> 
> If our core changes to the system can happen in devicemanager*.py, then we
> don't have to change much in our harnesses.  For Robocop I had to change
> harnesses and devicemanager*.py, I expect similar stuff here as well.

We won't run browser-chrome tests either, since we don't have any of the same chrome as Firefox.  I'm not sure whether it's worth trying to get the mochitest-chrome tests working or not; I'm guessing not if you're not using them for Fennec.

I'm attaching copies of b2gautomation.py (modeled after remoteautomation.py) and runtestsb2g.py (modeled after runtestsremote.py).  This works well locally for mochitest-plain, using the 'android.json' manifest of tests that have been vetted for Fennec.

Chunk 1/8 has these results:

I/GeckoDump( 2611): 21232 INFO TEST-START | Shutdown
I/GeckoDump( 2611): 21233 INFO Passed: 20587
I/GeckoDump( 2611): 21234 INFO Failed: 5
I/GeckoDump( 2611): 21235 INFO Todo:   211
I/GeckoDump( 2611): 21236 INFO SimpleTest FINISHED

The current code works well enough that it puts me in a position to be able to start working on the continuous integration bits, which is up next.

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: b2g mochitest automation support

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 618888 [details] [diff] [review]
b2g mochitest automation support

Review of attachment 618888 [details] [diff] [review]:
-----------------------------------------------------------------

Not too many nits, but a few.  As I mentioned in IRC, there is a lot of duplicated code.  I can't see an easy way to reuse the majority of the code without ugly hacks.  My goal is that when we are running these harnesses with mozbase, that can shed light on a better solution for reusing code.

::: build/mobile/b2gautomation.py
@@ +33,5 @@
> +# and other provisions required by the GPL or the LGPL. If you do not delete
> +# the provisions above, a recipient may use your version of this file under
> +# the terms of any one of the MPL, the GPL or the LGPL.
> +#
> +# ***** END LICENSE BLOCK *****

MPL 2.0?  I am not a contributor to this, except by reputation!

@@ +94,5 @@
> +        if crashreporter:
> +            env['MOZ_CRASHREPORTER_NO_REPORT'] = '1'
> +            env['MOZ_CRASHREPORTER'] = '1'
> +        else:
> +            env['MOZ_CRASHREPORTER_DISABLE'] = '1'

are these environment variables used in b2g?

@@ +119,5 @@
> +            args.remove('-foreground')
> +        except:
> +            pass
> +#TODO: figure out which platform require NO_EM_RESTART
> +#        return app, ['--environ:NO_EM_RESTART=1'] + args

since this is a unique file, you could remove this TODO.

@@ +156,5 @@
> +        serial = serial or self._devicemanager.deviceSerial
> +        status = 'unknown'
> +
> +        for line in self._devicemanager.runCmd(['devices']).stdout.readlines():
> +            result =  re.match('(.*?)\t(.*)', line)

this looks very blanket and potentially problematic.  No suggestios for something better.

::: testing/mochitest/runtestsb2g.py
@@ +32,5 @@
> +# and other provisions required by the GPL or the LGPL. If you do not delete
> +# the provisions above, a recipient may use your version of this file under
> +# the terms of any one of the MPL, the GPL or the LGPL.
> +#
> +# ***** END LICENSE BLOCK *****

MPL 2.0

Comment 5

[Jonathan Griffin (:jgriffin)]

I addressed the review comments and pushed as http://hg.mozilla.org/integration/mozilla-inbound/rev/acd7146a0484

Comment 6

[Ryan VanderMeulen [:RyanVM]]

http://hg.mozilla.org/mozilla-central/rev/acd7146a0484