904837 - Create jenkins automation for running firefoxos eideticker tests and uploading results

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

After discussing with :davehunt, I'm convinced that we should use Jenkins to download new builds of b2g/gaia, flash them onto a device, and then run eideticker tests on them. We should store the jenkins configuration files in revision control, so people can create their own setup (or hack on the existing one) if desired.

Likely we can reuse most of the work which has already been done for the existing datazilla tests here.

This bug is a prerequisite for creating a continuous integration type setup.

Comment 1

[William Lachance (:wlach)]

Setting magic tags so this will hopefully appear in perf team's sprint backlog.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Intitial working version is at https://github.com/davehunt/eideticker-ci however this will currently only attempt to run a single test and does not upload the results.

wlach: Could you try this out and provide feedback?

Comment 3

[William Lachance (:wlach)]

Attachment: run-update-dashboard-b2g.sh

I am mostly finished looking at this, got blocked with some issues in the harness which are not the fault of the jenkins automation. I will try to finish up my investigations tomorrow. In general it seems to work pretty well!

I would say we just need two things:

* Ability to actually update the dashboard with new data (see the attached script I was using for testing purposes as a base)
* Ability to configure the server we're uploading data to (see the ./bin/sync-dashboard.sh script)

Comment 4

[William Lachance (:wlach)]

Ok, so I'm still having troubles at the step where it tries to set the wifi network via gcli. Specifically I'm getting this error message:

12:19:43 + gcli setsetting screen.brightness 1
12:19:44 + gcli connectwifi --psk XXXXXXXX --key-management WPA-PSK "VIF WIFI 1"
12:19:44 usage: gcli [-h] [--address ADDRESS] <command> ...
12:19:44 gcli: error: unrecognized arguments: WIFI 1"

You mentioned earlier that the SSID should be the last argument to the utility. That makes sense and I made the correction, but I'm still seeing this problem. :( I'm not having the same problem running the latest version of gcli manually. Maybe it's an old copy of the utiltity being used by the ci framework?

Comment 5

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #4)
> Ok, so I'm still having troubles at the step where it tries to set the wifi
> network via gcli. Specifically I'm getting this error message:
> 
> 12:19:43 + gcli setsetting screen.brightness 1
> 12:19:44 + gcli connectwifi --psk XXXXXXXX --key-management WPA-PSK "VIF
> WIFI 1"
> 12:19:44 usage: gcli [-h] [--address ADDRESS] <command> ...
> 12:19:44 gcli: error: unrecognized arguments: WIFI 1"
> 
> You mentioned earlier that the SSID should be the last argument to the
> utility. That makes sense and I made the correction, but I'm still seeing
> this problem. :( I'm not having the same problem running the latest version
> of gcli manually. Maybe it's an old copy of the utiltity being used by the
> ci framework?

Note that when I run this on the command line, it works fine. Most strange...

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I have been able to replicate this, and working on a solution.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Fixed gcli to support open networks and WEP secured networks: https://github.com/davehunt/gcli/commit/7ecb43065edc712f009cbd955530b2b09d2fefa5

Fixed eideticker-ci to support WiFi networks with spaces in the SSID: https://github.com/davehunt/eideticker-ci/commit/2a7027d384c0703a61d4b4cdb8921f6d3ab2152c

Let me know if this fixes the issues you're having.

Comment 8

[Mike Lee [:mlee]]

Dave & William, any update on this?

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I have an instance of the CI working locally, and am just testing submitting the results to my people account. Once that's working I will request review of the system from Will. Unfortunately, whenever we run adhoc tests for partners it pushes this back.

Comment 10

[William Lachance (:wlach)]

Initial set of tests that Mike would definitely like to see (we can always do more, but this is the basic minimal set): contacts, clock, settings, camera, music

I believe we already have tests for all of these, so it's just a matter of making sure they're run.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The initial working version of the CI is now running on my machine, and reporting to http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/

You can check out the source code for the CI at https://github.com/davehunt/eideticker-ci

Can we close this bug, and raise new bugs for improvements such as moving it to a more permanent location?

Comment 12

[William Lachance (:wlach)]

(In reply to Dave Hunt (:davehunt) from comment #11)
> Can we close this bug, and raise new bugs for improvements such as moving it
> to a more permanent location?

Makes sense to me.