990635 - [marionette-js-runner] Create host for connecting to python harness over a socket

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::TestAgent, defect, P1)

Description

[Andrew Halberstadt [:ahal]]

We'd like to be able to re-use most of our python test harness stack when running gaia-integration tests in tbpl. To do this we should be able to create (or modify an existing) host that:

1. sends a profile path over a socket
2. waits to receive a test to run
3. runs test
4. sends back result
5. go to 2 and repeat

The python side would be responsible for starting b2gdesktop (or device), and doing any necessary cleanup between tests.

Comment 1

[Gareth Aye [:gaye] (back from PTO)]

Wouldn't we go back to step 1 and send over a new python path? eg https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L68

Comment 2

[Andrew Halberstadt [:ahal]]

I assumed we built one profile per test run, but I guess individual tests require specialized profiles?

Comment 3

[Andrew Halberstadt [:ahal]]

I started working on the transport piece the last day or so. Though the focus of it is this use case for now, I want the transport to be re-usable by any future test harnesses which might want:
A) bindings across multiple languages, or
B) the ability to easily distribute tests across threads, processes or machines (or combinations of 

The service is: https://github.com/ahal/corredor
The JS client for it is: https://github.com/ahal/corredor-js-client

Right now the prototype simulates sending and running tests. To try it out locally you need zmq:
libzmq - apt-get install libzmq3-dev, yum install zeromq3-devel, brew install zmq
python bindings - pip install pyzmq
node bindings - npm install zmq

Then clone both repos and in separate terminals (and either order), run:
python test_program.py
node test_worker.js

The python side sends tests to the js side one at a time and receives the result. It also receives all stdout and stderr from the js side via a separate socket.

Comment 4

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
> I assumed we built one profile per test run, but I guess individual tests
> require specialized profiles?

Yep!

Comment 5

[Andrew Halberstadt [:ahal]]

I have a very hacky prototype almost working. I'm still not sure where everything will go, but so far I've created new branches on the related projects:

https://github.com/ahal/gaia/tree/python_service
https://github.com/ahal/marionette-b2gdesktop-host/tree/python_service
https://github.com/ahal/mocha-tbpl-reporter/tree/python_service

I hope to have tests running sometime early next week. A couple things to note.. I am a complete node newb (and my javascript skills in general are quite rusty). So my hope is for you to use my hacky code as an example only and to help me package every thing properly (to get this working locally I created a jumble of symlinks).

Also feedback on the transport piece (e.g corredor-js-client) is more than welcome! It will still likely change a lot before this is all done. Speaking of which, I guess I'll need to register it on npm at some point?

Comment 6

[Andrew Halberstadt [:ahal]]

So I have one test running successfully, but I'm running into some trouble trying to get the next one to start.

This is output from an *unmodified* setup:

START RUNNER
baz
    ✓ is running and listening 
STOP RUNNER
START RUNNER
STOP RUNNER

  Music player tests
START RUNNER
baz
    Audio channels tests
      ✓ Interrupted by a higher priority channel (14717ms)
STOP RUNNER
START RUNNER
STOP RUNNER

  Awesomescreen

Gareth, do you know what's up with the extra "START/STOP" call after each suite? I put those debug statements here: https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L68

That in-between call is tripping up the ordering of events across the socket, and I'm having difficulty understanding its purpose or where it's coming from.

Comment 7

[Andrew Halberstadt [:ahal]]

I discovered it's not the ordering that was causing me problems, but I'm still curious about the question.

Comment 8

[Gareth Aye [:gaye] (back from PTO)]

Interesting... I don't think this is happening in marionette-js-runner proper, but I am meaning to play around with your code later this week. I'll probably have a good diagnosis for you then.

Comment 9

[Andrew Halberstadt [:ahal]]

That output is from marionette-js-runner proper (I just added a few console.logs). I think what is happening is that the callback from stop() always calls start() again, even if it's the last test in the suite. Then suiteTeardown calls stop() one last time.

Comment 10

[Andrew Halberstadt [:ahal]]

This brings me to another question though. I noticed that each suite creates a new Host instance. Do we want to support multiple Host's all connected to the same socket endpoint at the same time? If yes, then I'll need to change the transport a bit as I currently assume a 1-1 connection (this shouldn't be a big deal). Other than that, I have tests running on my local machine.

My goal is to have things ready for you to look at by the end of the week. I still need to handle output, results and make things less hacky.

Comment 11

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Andrew Halberstadt [:ahal] from comment #10)
> This brings me to another question though. I noticed that each suite creates
> a new Host instance. Do we want to support multiple Host's all connected to
> the same socket endpoint at the same time?

I don't have a strong use case for this right now. Not to say that there's no reason to do this... just that I don't [personally] have a user story :).

> If yes, then I'll need to change
> the transport a bit as I currently assume a 1-1 connection (this shouldn't
> be a big deal). Other than that, I have tests running on my local machine.

Sweet!

> My goal is to have things ready for you to look at by the end of the week. I
> still need to handle output, results and make things less hacky.

Cool I'm starting to read now.

Comment 12

[Gareth Aye [:gaye] (back from PTO)]

Hey if you need a reviewer for some of this stuff I am happy to get to know the codebase

Comment 13

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Right now the prototype simulates sending and running tests. To try it out
> locally you need zmq:
> libzmq - apt-get install libzmq3-dev, yum install zeromq3-devel, brew
> install zmq
> python bindings - pip install pyzmq
> node bindings - npm install zmq

I think we ask gaia developers to install a bunch of stuff locally and it will be important for us to be able to run this locally without too much friction. Is there any way that we can package zmq in a way that's less global?

Comment 14

[Gareth Aye [:gaye] (back from PTO)]

Also for language bindings, you should add package.json and requirements.txt so that you can specify binding versions and install local to project

Comment 15

[Andrew Halberstadt [:ahal]]

So for the python/node bindings we can just have pip/npm install them as packages. For the library we can add it to the Firefox OS build prereqs and the mach bootstrap command. Alternatively, the python zmq binding will attempt to compile and install the library from source if it's not present, we could just let it attempt this.

The way I envision it whatever entry point we use (e.g a mach command) would be responsible for making sure this is all set up. It can install the python stuff in a virtualenv to avoid global packages (not sure if node has something similar or not).

Also I filed bug 994764 as a top level tracking bug for all this. I was thinking we could use this specifically for the js host part.

Comment 16

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Andrew Halberstadt [:ahal] from comment #15)

> The way I envision it whatever entry point we use (e.g a mach command) would
> be responsible for making sure this is all set up. It can install the python
> stuff in a virtualenv to avoid global packages (not sure if node has
> something similar or not).

That sounds good. In node/npm require will find things installed in node_modules before it looks at globally installed packages.

> Also I filed bug 994764 as a top level tracking bug for all this. I was
> thinking we could use this specifically for the js host part.

Sure sounds good.

Comment 17

[Gareth Aye [:gaye] (back from PTO)]

corredor-js-client/strategy.js:19
  this.master.socket.actionMap[action] = callback;
                                       ^
TypeError: Cannot set property 'test_start' of undefined
    at PingPong.registerAction (/Users/gareth/Documents/corredor-js-client/strategy.js:19:40)
    at Object.<anonymous> (/Users/gareth/Documents/corredor-js-client/test_worker.js:17:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

yikes

Comment 18

[Andrew Halberstadt [:ahal]]

Oh, I changed things around a bit since comment 3. I haven't been using test_worker.js anymore since I've got actual tests running now :). I just forgot to delete it.

To try running tests locally you can:
1) use the 'python_service' branch of gaia, marionette-b2gdesktop-host and mocha-tbpl-reporter on my github. (I just linked them all to ~/.node_modules since I had no idea how to get them pulled in via the Makefile)

2) install corredor (preferably in a virtualenv):
$ cd path/to/corredor
$ python setup.py develop

3) Run the python runner:
$ cd path/to/gaia
$ python tests/python/runner-service/runner.py

Sorry, as you can see I'm still trying to sort all this out :).

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Host for sending start/stop commands over a socket

Link: https://github.com/ahal/marionette-socket-host

Gareth, could you also give this a quick look? It's the same idea as the last time. I copied marionette-b2gdesktop-host and changed around what was needed.

Comment 21

[Gareth Aye [:gaye] (back from PTO)]

Hey :ahal. I have a few thoughts...

(1) https://github.com/ahal/marionette-socket-host/blob/master/package.json#L15 lock down dependencies so we don't get burned by breaking, upstream api changes
(2) promises over callbacks in corredor since that's what much of gaia is moving toward (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise supported in node >= 0.11 and there's a polyfill for node <0.11 here https://github.com/jakearchibald/es6-promise)
(3) If you want developers to run these when and if tests break, you'd have much better luck selling them on it if they didn't have to install zeromq. Maybe linux container something?
(4) Stuff got copied over from b2g desktop host that's not being used (eg https://github.com/ahal/marionette-socket-host/blob/master/index.js#L40) and some comments (eg https://github.com/ahal/marionette-socket-host/blob/master/index.js#L11) are outdated
(5) The debug statements could be much more useful
(6) Small nit https://github.com/ahal/marionette-socket-host/blob/master/test/index.js should be named somethingorother_test.js or test_somethingorother.js.
(7) https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L68 is a subject in the scope of that function block that "shadows" this one https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L21
(8) https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L57 seems like you call done synchronously here

I have many more nits, but I hope that can get you started in the right direction. Let me know if you have any questions and thanks for the help!

Comment 22

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8453383 [details]
Host for sending start/stop commands over a socket

Reflag if you'd like another review pass later!

Comment 23

[Andrew Halberstadt [:ahal]]

(In reply to Gareth Aye [:gaye] from comment #21)
> Hey :ahal. I have a few thoughts...

Thanks, this is helpful!

> (1)
> https://github.com/ahal/marionette-socket-host/blob/master/package.json#L15
> lock down dependencies so we don't get burned by breaking, upstream api
> changes

Even "debug"? mozilla-b2g/gaia-node-modules doesn't peg it either, so I think requiring a specific version here might cause problems in automation?

> (2) promises over callbacks in corredor since that's what much of gaia is
> moving toward
> (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/
> Global_Objects/Promise supported in node >= 0.11 and there's a polyfill for
> node <0.11 here https://github.com/jakearchibald/es6-promise)

This seems like a good idea. I'll file a follow up for it though. My immediate goal is to get what I have now landed by next week so others can start playing around with it.

> (3) If you want developers to run these when and if tests break, you'd have
> much better luck selling them on it if they didn't have to install zeromq.
> Maybe linux container something?

Agreed. Pyzmq will attempt to download the source and compile it for you. Barring that, I can also get whatever will be running this (e.g mach) to do the necessary setup. Again though, I'll do this in a follow-up.

> (8)
> https://github.com/ahal/marionette-socket-host/blob/master/test/index.js#L57
> seems like you call done synchronously here

Isn't child_process.fork synchronous? At least I don't see any events that might signal when the fork is ready.. Either way, in this case it doesn't matter. ZeroMQ makes no assumptions about the ordering of calls to bind/connect. It will buffer messages in the event there is nothing on the other end and deliver them as soon as something connects.

> I have many more nits, but I hope that can get you started in the right
> direction. Let me know if you have any questions and thanks for the help!

Cool, I pushed the fixes to my repo minus some comments I replied to here.

Comment 24

[Andrew Halberstadt [:ahal]]

Comment on attachment 8453383 [details]
Host for sending start/stop commands over a socket

If you have other nits, I'd love to hear them.

Comment 25

[Andrew Halberstadt [:ahal]]

I transferred this to mozilla-b2g: https://github.com/mozilla-b2g/marionette-socket-host

Need someone to add the chefs list for commit access. Will wait for that and additional review comments before resolving this.

Comment 26

[Andrew Halberstadt [:ahal]]

I'm declaring this done. We can file new bugs for further fixes.

Final repo is here: https://github.com/mozilla-b2g/marionette-socket-host