994764 - [Tracking] Gaia integration tests using python toolchain over a socket

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::TestAgent, defect, P1)

Description

[Andrew Halberstadt [:ahal]]

This bug tracks progress towards using the standard automated testing toolchain with gaia integration tests.

Doing this will allow us to re-use components for starting/stopping gecko on various platforms, crash detection, result reporting, mozharness integration, etc.

There are several pieces to this. First a transport for carrying messages will be needed:
https://github.com/ahal/corredor
https://github.com/ahal/corredor-js-client

Next we'll need to create/modify a host target for marionette-js-runner:
https://github.com/ahal/marionette-b2gdesktop-host/tree/python_service

Then we'll need to create/modify a mocha reporter that streams results:
https://github.com/ahal/mocha-tbpl-reporter/tree/python_service

Finally we need the python service that does the work:
https://github.com/ahal/gaia/tree/python_service

Note that all of these repos/branches are just a prototype. There is a good chance they will be deleted/moved/merged around to new locations. I'll file individual bugs for each piece to keep better track of them.

Comment 1

[Andrew Halberstadt [:ahal]]

Oh, I also forgot to mention I needed a minor change to marionette-js-runner itself to differentiate between test-end and suite-end in the host:
https://github.com/ahal/marionette-js-runner/commit/e2b1d794c850c1deb6def5547504faa0a283fa11

Comment 2

[Andrew Halberstadt [:ahal]]

Just a quick progress update since it's been quiet for awhile.

I've been working on bug 994847 most of this time. Basically why that is needed is because our tool for starting and stopping gecko processes (mozrunner) was created pre-b2g, and so wasn't really designed with remote devices in mind. To move quickly, we instead placed the code for starting/stopping the emulator in the python marionette_client (even though it conceptually should have been in mozrunner).

Now that we want to run Gi on emulators as well as devices, it feels exceptionally awkward to use the python marionette_client to run the JS marionette_client. So I'm shuffling a bunch of code around so everything related to starting and stopping things lives in mozrunner. In addition to being cleaner and easier to maintain, this work will make it much easier for us to expand to devices as well.

Comment 3

[Andrew Halberstadt [:ahal]]

Actually, I meant to link to bug 997244 (the above bug's dependent).

Comment 5

[Andrew Halberstadt [:ahal]]

Quick progress update: Gi seems to be running and mostly passing on device (and probably emulator too, but that's untested as of yet) for me locally.

There are a ton of moving pieces and different repos involved, so my next priority is probably going to be figuring out how to set everything up so other people can easily test it out locally as well. I think this will mostly be creating new npm packages and adding them to https://github.com/mozilla-b2g/gaia-node-modules. I'll also need to land a series of patches against mozbase.

I think smoothing this out and getting these pieces landed so that it is possible to run most of the suite locally is achieve-able in the next two weeks. Getting all the tests to pass, adding all the bells and whistles and getting it stable enough for continuous integration is a much bigger task.

Comment 6

[Andrew Halberstadt [:ahal]]

Most of the pieces are landed. The one notable exception is bug 1037089 which is blocked by bug 1042183.

There is still a lot of work to do. Most of the work falls into a few buckets:
* Adding all the test harness bells and whistles (things like crash reporting)
* Making it easy to run out of the box (in automation and locally for devs)
* Performance improvements (figure out a way to not reboot the device each time?)

If anyone wants to try this at home, here are the steps you need to do:

1) Install libzmq, one of:
$ sudo apt-get install libzmq3-dev
$ sudo yum install zeromq3-devel
$ sudo brew install zmq

2) Install runner-service:
$ cd gaia/tests/python/runner-service
$ python setup.py install

3) Install npm packages
$ npm install marionette-socket-host
$ npm install mocha-socket-reporter

4) Connect device and run:
$ gaia-integration --buildapp device

The plan is to condense all those steps into a single command that just works, whether we use mach or just create a new make target for now.

Comment 7

[Andrew Halberstadt [:ahal]]

Note: step 3 should be performed from the root gaia directory. You may also need to run "make node_modules" first.

Comment 8

[Bob Silverberg [:bsilverberg]]

I'm not sure why, but I had to run the python install with an environment variable set, like so:

ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future python setup.py install

I did this because I saw an error message (clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]) that was similar to one I've seen before when trying to install the Python MySQL package.

Documenting this for anyone else who tries and encounters a similar issue.

Comment 9

[Bob Silverberg [:bsilverberg]]

The procedure in comment 6 is failing for me at step 3. I tried running `make node_modules` first, and that yielded: 

https://gist.github.com/bobsilverberg/b9907d58dafaeeaec976

Thinking that maybe I could just get away with running `npm install marionette-socket-host`, I tried that, which yielded:

https://gist.github.com/bobsilverberg/b14aeaea5a51a050b485

:mdas, I understand from stephend that you are now supporting this effort. Can you help me get this up and running locally?

Comment 10

[Jonathan Griffin (:jgriffin)]

The problem you're running into occurs because the precompiled modules available from 'make node_modules' are for linux, and you're installing this on the Mac.  Normally that could work, but there are some libraries on your machine which are out-of-date:

  Reason: Incompatible library version: cc1plus requires version 13.0.0 or later, but libgmp.10.dylib provides version 11.0.0

If I had to guess, the development libs on your machine are from an earlier version of the OS, and you've since upgraded to OSX 10.9.

Following these instructions _might_ resolve this problem:  https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Mac_OS_X_Prerequisites

Comment 11

[Bob Silverberg [:bsilverberg]]

I tried that, following both the "One-Line Setup" and the more detailed instructions, but nether seems to have helped with that Incompatible library version error. I'm not sure what to do next.

Comment 12

[Bob Silverberg [:bsilverberg]]

I am still on OS X v 10.8.5, which I believe is the version it came with.

Comment 13

[Bob Silverberg [:bsilverberg]]

Luckily I have a second machine which has OS X 10.9.2 and I was able to complete the install steps on that. I'm having trouble running the tests, but I'm going to see if zac can help me with that.

Comment 14

[Bob Silverberg [:bsilverberg]]

I have tried running the tests using the command `gaia-integration --buildapp device` and have encountered a couple of problems:

1. If I do not specify a manifest file then it tries to find the file in the wrong place:

IOError: Missing files: /Users/bsilverberg_local/.virtualenvs/gaia-js-runner/lib/shared/test/integration/manifest.ini

It's looking for the file in my virtual env rather than in the gaia folder.

2. If I point it to the correct path using `--manifest` then I don't get an error message, but all I see in the console is:

make: *** No rule to make target `test-integration-test'.  Stop.

and it just sits there.  If I ^C out of that, I see: https://gist.github.com/bobsilverberg/c444f94c404fe833950d

Zac is able to get the tests running on his machine but I am not.

:mdas or :ahal, are either of you able to help me work through this issue?

Comment 15

[Andrew Halberstadt [:ahal]]

The second problem is the same as the first. To get things working quickly I put in an awful hack:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/runner-service/runner_service/runintegration.py#L18

I think what happened is you did "setup.py install" and zac did "setup.py develop" (which doesn't copy the file over to a virtualenv). Instead of hardcoding it like that, we should walk down the ancestor directories from 'cwd' until we find it.

So to unblock yourself in the short term, try doing setup.py develop instead.

Comment 16

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Andrew Halberstadt [:ahal] from comment #15)
> The second problem is the same as the first. To get things working quickly I
> put in an awful hack:
> https://github.com/mozilla-b2g/gaia/blob/master/tests/python/runner-service/
> runner_service/runintegration.py#L18
> 
> I think what happened is you did "setup.py install" and zac did "setup.py
> develop" (which doesn't copy the file over to a virtualenv). Instead of
> hardcoding it like that, we should walk down the ancestor directories from
> 'cwd' until we find it.
> 
> So to unblock yourself in the short term, try doing setup.py develop instead.

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1050446 to fix this

Comment 17

[Bob Silverberg [:bsilverberg]]

Thanks :ahal. That worked for me and got me past that point. I am now getting a different error message after all of the downloading and setup take place: https://gist.github.com/bobsilverberg/3074c89376e219e0eda9

That is the output of the 2nd time I ran gaia-integration. The first time included all of the downloading and setup output which I thought you might not need.

As it sounds like a node module might be missing, I tried running both `npm install marionette-socket-host` and `npm install mocha-socket-reporter` but they both seem ok. Output: https://gist.github.com/bobsilverberg/5e2ff38aae41afd90190

Let me know what else I can try to get past this next roadblock.

Comment 18

[Andrew Halberstadt [:ahal]]

I think I saw that error too once (it's unrelated to this bug). I don't remember why it was happening, but I'd guess "npm install sockit-to-me" should get you passed it.

Comment 19

[Bob Silverberg [:bsilverberg]]

Thanks :ahal. That worked. Here is the next error that happens after resolving that:

https://gist.github.com/bobsilverberg/3d83f0faea9e4e19296e

Comment 20

[Andrew Halberstadt [:ahal]]

Can you run "make clean && make really-clean && make gaia-integration" without errors?

I'm not sure what is up with that, but maybe we should take this out of the bug. I think these errors are more related to the gaia build system/make targets than the python runner service. Feel free to email or ping me on irc!

Comment 21

[Andrew Halberstadt [:ahal]]

Oh sorry, that last one should be "make test-integration"

Comment 22

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Andrew Halberstadt [:ahal] from comment #6)
> Most of the pieces are landed. The one notable exception is bug 1037089
> which is blocked by bug 1042183.
> 
> There is still a lot of work to do. Most of the work falls into a few
> buckets:
> * Adding all the test harness bells and whistles (things like crash
> reporting)
> * Making it easy to run out of the box (in automation and locally for devs)
> * Performance improvements (figure out a way to not reboot the device each
> time?)
> 
> If anyone wants to try this at home, here are the steps you need to do:
> 
> 1) Install libzmq, one of:
> $ sudo apt-get install libzmq3-dev
> $ sudo yum install zeromq3-devel
> $ sudo brew install zmq
>

as a note, zmq version 4.0.4 is needed for the python libs to work in OS X (Mavericks at least). I had to run 'brew update' to make sure I got the most recent version.

Comment 23

[Bob Silverberg [:bsilverberg]]

:ahal, I agree that this bug is probably not the best place for this. Perhaps we should have a new bug for it as it is important to track. I can contact you via email next week, and maybe we can even get together in person to work on it. I am in Toronto but usually work from home. Are you based in Toronto as well?

FTR, I cannot successfully run "make clean && make really-clean && make test-integration" (output at [1]), so I guess it's a build issue, but I'm not sure why we need to do all this building in order to just run tests on a device which already has everything loaded on it. I am assuming it's part of the process of trying to get this working, and the final solution won't require any of this building (we don't need to do that for the Python integration tests). If that is correct, I wonder if that's something that can be accomplished in the near term?

[1] https://gist.github.com/bobsilverberg/2b40bf7058d2fd92c32c

Comment 24

[Julien Wajsberg [:julienw]]

Just to give some context, "make clean" is currently necessary if you made some code change. If you only changed a test file, then

  TEST_FILES="file names" bin/gaia-marionette

works fine.

In your case, the error is quite strange, because ^1.2.11 should find 1.2.11 which it says is available.

You can try to quickfix this by editing package.json and replacing "^1.2.11" at the end by "1.2.11". You can also check your npm version (I use npm 1.4.3 and node 1.10.26, which are not really current).

Comment 25

[Andrew Halberstadt [:ahal]]

(In reply to Bob Silverberg [:bsilverberg] from comment #23)
> FTR, I cannot successfully run "make clean && make really-clean && make
> test-integration" (output at [1]), so I guess it's a build issue, but I'm
> not sure why we need to do all this building in order to just run tests on a
> device which already has everything loaded on it. I am assuming it's part of
> the process of trying to get this working, and the final solution won't
> require any of this building (we don't need to do that for the Python
> integration tests). If that is correct, I wonder if that's something that
> can be accomplished in the near term?
> 
> [1] https://gist.github.com/bobsilverberg/2b40bf7058d2fd92c32c

It's not really "building", it's installing dependencies, and yes we'll need to continue doing that as the marionette-js-runner and other various gaia libs depend on them. We won't need to download b2g_desktop however (I think there is a bug on file to stop doing this already).

Comment 26

[Bob Silverberg [:bsilverberg]]

Thanks :ahal. I was referring to commands like `run-js-command email/app/build`, which I took to be building the apps. If we're running on a device with a pre-existing build, why do we need to do that?

:jukienw, I'll give that a try.