Create a Jenkins job for experimenting with gaia-integration tests

RESOLVED WONTFIX

Status

Firefox OS
Infrastructure
RESOLVED WONTFIX
4 years ago
5 months ago

People

(Reporter: jgriffin, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Per bug 1042950, this job should be run against nodes 16/17, since those are the ones that have had libzmq installed.

Ahal, can you provide instructions for how to invoke the tests?
Sure (copied from https://bugzilla.mozilla.org/show_bug.cgi?id=994764#c6):

If anyone wants to try this at home, here are the steps you need to do:

1) Install libzmq, one of:
$ sudo apt-get install libzmq3-dev
$ sudo yum install zeromq3-devel
$ sudo brew install zmq

2) Install runner-service:
$ cd gaia/tests/python/runner-service
$ python setup.py install

3) Install npm packages
$ cd <root gaia dir>
$ npm install marionette-socket-host
$ npm install mocha-socket-reporter

4) Connect device and run:
$ gaia-integration --buildapp device

The plan is to condense all those steps into a single command that just works, whether we use mach or just create a new make target for now.
Thanks! We're currently at a work week right now, but we will get to it when we can. I've updated the reference bug such that we will run these on nodes 23/24.
So I'm having trouble setting up the job mostly as I'm not sure what flags gaia-integration takes in. Does it allow for specifying the serial number of the device for the machines that have more than one device on them?
Flags: needinfo?(ahalberstadt)
(In reply to Dylan Chun Wong [:dwong] from comment #3)
> So I'm having trouble setting up the job mostly as I'm not sure what flags
> gaia-integration takes in. Does it allow for specifying the serial number of
> the device for the machines that have more than one device on them?

No not yet, but that should be easy to add! I filed bug 1046713 for it. This is why it's good to get other people to try things out, to catch stuff like this :).

Fyi, the code that runs this lives here if you wanted to take a look: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/runner-service (runintegration.py is the entry point).
Flags: needinfo?(ahalberstadt)
(In reply to Dylan Chun Wong [:dwong] from comment #3)
> So I'm having trouble setting up the job mostly as I'm not sure what flags
> gaia-integration takes in. Does it allow for specifying the serial number of
> the device for the machines that have more than one device on them?

Dylan, while we await the fix for bug 1046713, are you able to set up a job to run a simple JS test or two, on-device, using the above setup in comment 1, etc.?  Thx :-)
(Bob, Zac, Dave, et al -- if you'd like to dive in too, please do!)
Flags: needinfo?(dwong)
I can try, it would just need to be on one of the nodes that have a single device like 15.
Flags: needinfo?(dwong)

Comment 7

4 years ago
IMO let's validate it all locally with a few test runs before we run it on Jenkins. If it leaves the device in a bad state or sometihng then we could have a lot of hassle on the ground in MV.

Comment 8

4 years ago
Re comment #7, I've just been running the tests locally and there is not much point running them on CI just yet. There are a number of different failures that need different fixes, running on CI would just create noise.

Comment 9

4 years ago
This is all blocked by most of https://bugzilla.mozilla.org/show_bug.cgi?id=1034504

and then a few more issues where specific tests need to be updated to deal with the difference with device running. There's not much point in running on CI because it's difficult to debug detail problems - it's better to have the device in your hand.

We also need a way to just run single tests while we're debugging these issues (not sure if this is raised, I'm sure ahal is 'on it')
Hi I was just wondering what version of node is needed to run these tests? I was on v0.11.14-pre, but it seems that node-gyp doesn't play nice with it.
Flags: needinfo?(ahalberstadt)
Hm, I'm not really sure. All I can say for sure is that I'm using 0.10.29, and that works for me.
Flags: needinfo?(ahalberstadt)
Jenkins has 0.10.28, which is will install for any job with the "Provide Node & npm bin/ folder to PATH" option checked. If you need a different version then we can add it, but I suspect this should be fine.
Ah, I see. I was just playing around with trying to get it working locally on my own machine, as well as Jenkins. I didn't know about the option, but now I do!
Hi I just wanted to get tests to run at least one on Jenkins and was able to get it to run at least once: http://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.gaia-integration/15/console

My trouble was with node-gyp, but reinstalling node to a stable version and by removing everything node and npm related it worked. And also replacing 'python setup.py install' to 'python setup.py develop' as mentioned in another tracking bug. As brought up by zac it left it in a bad state. Stuck on a black screen, but was able to be put into fastboot mode and restart it. This was just to make sure I could install all the necessary libraries and run it on Jenkins. The job isn't set to run automatically as of now.
Component: WebQA → Infrastructure
Product: Testing → Firefox OS

Comment 15

5 months ago
Firefox OS is not being worked on
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.