1046264 - Create a Jenkins job for experimenting with gaia-integration tests

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Infrastructure, defect)

Description

[Jonathan Griffin (:jgriffin)]

Per bug 1042950, this job should be run against nodes 16/17, since those are the ones that have had libzmq installed.

Ahal, can you provide instructions for how to invoke the tests?

Comment 1

[Andrew Halberstadt [:ahal]]

Sure (copied from https://bugzilla.mozilla.org/show_bug.cgi?id=994764#c6):

If anyone wants to try this at home, here are the steps you need to do:

1) Install libzmq, one of:
$ sudo apt-get install libzmq3-dev
$ sudo yum install zeromq3-devel
$ sudo brew install zmq

2) Install runner-service:
$ cd gaia/tests/python/runner-service
$ python setup.py install

3) Install npm packages
$ cd <root gaia dir>
$ npm install marionette-socket-host
$ npm install mocha-socket-reporter

4) Connect device and run:
$ gaia-integration --buildapp device

The plan is to condense all those steps into a single command that just works, whether we use mach or just create a new make target for now.

Comment 2

[Dylan Chun Wong [:dwong]]

Thanks! We're currently at a work week right now, but we will get to it when we can. I've updated the reference bug such that we will run these on nodes 23/24.

Comment 3

[Dylan Chun Wong [:dwong]]

So I'm having trouble setting up the job mostly as I'm not sure what flags gaia-integration takes in. Does it allow for specifying the serial number of the device for the machines that have more than one device on them?

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to Dylan Chun Wong [:dwong] from comment #3)
> So I'm having trouble setting up the job mostly as I'm not sure what flags
> gaia-integration takes in. Does it allow for specifying the serial number of
> the device for the machines that have more than one device on them?

No not yet, but that should be easy to add! I filed bug 1046713 for it. This is why it's good to get other people to try things out, to catch stuff like this :).

Fyi, the code that runs this lives here if you wanted to take a look: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/runner-service (runintegration.py is the entry point).

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Dylan Chun Wong [:dwong] from comment #3)
> So I'm having trouble setting up the job mostly as I'm not sure what flags
> gaia-integration takes in. Does it allow for specifying the serial number of
> the device for the machines that have more than one device on them?

Dylan, while we await the fix for bug 1046713, are you able to set up a job to run a simple JS test or two, on-device, using the above setup in comment 1, etc.?  Thx :-)
(Bob, Zac, Dave, et al -- if you'd like to dive in too, please do!)

Comment 6

[Dylan Chun Wong [:dwong]]

I can try, it would just need to be on one of the nodes that have a single device like 15.

Comment 7

[Zac C (:zac)]

IMO let's validate it all locally with a few test runs before we run it on Jenkins. If it leaves the device in a bad state or sometihng then we could have a lot of hassle on the ground in MV.

Comment 8

[Zac C (:zac)]

Re comment #7, I've just been running the tests locally and there is not much point running them on CI just yet. There are a number of different failures that need different fixes, running on CI would just create noise.

Comment 9

[Zac C (:zac)]

This is all blocked by most of https://bugzilla.mozilla.org/show_bug.cgi?id=1034504

and then a few more issues where specific tests need to be updated to deal with the difference with device running. There's not much point in running on CI because it's difficult to debug detail problems - it's better to have the device in your hand.

We also need a way to just run single tests while we're debugging these issues (not sure if this is raised, I'm sure ahal is 'on it')

Comment 10

[Dylan Chun Wong [:dwong]]

Hi I was just wondering what version of node is needed to run these tests? I was on v0.11.14-pre, but it seems that node-gyp doesn't play nice with it.

Comment 11

[Andrew Halberstadt [:ahal]]

Hm, I'm not really sure. All I can say for sure is that I'm using 0.10.29, and that works for me.

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Jenkins has 0.10.28, which is will install for any job with the "Provide Node & npm bin/ folder to PATH" option checked. If you need a different version then we can add it, but I suspect this should be fine.

Comment 13

[Dylan Chun Wong [:dwong]]

Ah, I see. I was just playing around with trying to get it working locally on my own machine, as well as Jenkins. I didn't know about the option, but now I do!

Comment 14

[Dylan Chun Wong [:dwong]]

Hi I just wanted to get tests to run at least one on Jenkins and was able to get it to run at least once: http://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.gaia-integration/15/console

My trouble was with node-gyp, but reinstalling node to a stable version and by removing everything node and npm related it worked. And also replacing 'python setup.py install' to 'python setup.py develop' as mentioned in another tracking bug. As brought up by zac it left it in a bad state. Stuck on a black screen, but was able to be put into fastboot mode and restart it. This was just to make sure I could install all the necessary libraries and run it on Jenkins. The job isn't set to run automatically as of now.

Comment 15

[BMO Automation]

Firefox OS is not being worked on