Closed Bug 904837 Opened 8 years ago Closed 8 years ago

Create jenkins automation for running firefoxos eideticker tests and uploading results

Categories

(Testing Graveyard :: Eideticker, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wlach, Assigned: davehunt)

References

Details

(Keywords: perf, Whiteboard: [c=automation p= s= u=])

Attachments

(1 file)

After discussing with :davehunt, I'm convinced that we should use Jenkins to download new builds of b2g/gaia, flash them onto a device, and then run eideticker tests on them. We should store the jenkins configuration files in revision control, so people can create their own setup (or hack on the existing one) if desired.

Likely we can reuse most of the work which has already been done for the existing datazilla tests here.

This bug is a prerequisite for creating a continuous integration type setup.
Setting magic tags so this will hopefully appear in perf team's sprint backlog.
Keywords: perf
Whiteboard: [c= p=]
Intitial working version is at https://github.com/davehunt/eideticker-ci however this will currently only attempt to run a single test and does not upload the results.

wlach: Could you try this out and provide feedback?
Flags: needinfo?(wlachance)
Blocks: 916989
Status: NEW → ASSIGNED
Whiteboard: [c= p=] → [c=automation p= s= u=]
I am mostly finished looking at this, got blocked with some issues in the harness which are not the fault of the jenkins automation. I will try to finish up my investigations tomorrow. In general it seems to work pretty well!

I would say we just need two things:

* Ability to actually update the dashboard with new data (see the attached script I was using for testing purposes as a base)
* Ability to configure the server we're uploading data to (see the ./bin/sync-dashboard.sh script)
Attachment #805732 - Attachment mime type: application/x-shellscript → text/plain
Ok, so I'm still having troubles at the step where it tries to set the wifi network via gcli. Specifically I'm getting this error message:

12:19:43 + gcli setsetting screen.brightness 1
12:19:44 + gcli connectwifi --psk XXXXXXXX --key-management WPA-PSK "VIF WIFI 1"
12:19:44 usage: gcli [-h] [--address ADDRESS] <command> ...
12:19:44 gcli: error: unrecognized arguments: WIFI 1"

You mentioned earlier that the SSID should be the last argument to the utility. That makes sense and I made the correction, but I'm still seeing this problem. :( I'm not having the same problem running the latest version of gcli manually. Maybe it's an old copy of the utiltity being used by the ci framework?
Flags: needinfo?(dave.hunt)
Flags: needinfo?(wlachance)
(In reply to William Lachance (:wlach) from comment #4)
> Ok, so I'm still having troubles at the step where it tries to set the wifi
> network via gcli. Specifically I'm getting this error message:
> 
> 12:19:43 + gcli setsetting screen.brightness 1
> 12:19:44 + gcli connectwifi --psk XXXXXXXX --key-management WPA-PSK "VIF
> WIFI 1"
> 12:19:44 usage: gcli [-h] [--address ADDRESS] <command> ...
> 12:19:44 gcli: error: unrecognized arguments: WIFI 1"
> 
> You mentioned earlier that the SSID should be the last argument to the
> utility. That makes sense and I made the correction, but I'm still seeing
> this problem. :( I'm not having the same problem running the latest version
> of gcli manually. Maybe it's an old copy of the utiltity being used by the
> ci framework?

Note that when I run this on the command line, it works fine. Most strange...
I have been able to replicate this, and working on a solution.
Flags: needinfo?(dave.hunt)
Fixed gcli to support open networks and WEP secured networks: https://github.com/davehunt/gcli/commit/7ecb43065edc712f009cbd955530b2b09d2fefa5

Fixed eideticker-ci to support WiFi networks with spaces in the SSID: https://github.com/davehunt/eideticker-ci/commit/2a7027d384c0703a61d4b4cdb8921f6d3ab2152c

Let me know if this fixes the issues you're having.
Dave & William, any update on this?
Flags: needinfo?(wlachance)
Flags: needinfo?(dave.hunt)
I have an instance of the CI working locally, and am just testing submitting the results to my people account. Once that's working I will request review of the system from Will. Unfortunately, whenever we run adhoc tests for partners it pushes this back.
Flags: needinfo?(dave.hunt)
Initial set of tests that Mike would definitely like to see (we can always do more, but this is the basic minimal set): contacts, clock, settings, camera, music

I believe we already have tests for all of these, so it's just a matter of making sure they're run.
Flags: needinfo?(wlachance)
Depends on: 917404
Depends on: 929987
The initial working version of the CI is now running on my machine, and reporting to http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/

You can check out the source code for the CI at https://github.com/davehunt/eideticker-ci

Can we close this bug, and raise new bugs for improvements such as moving it to a more permanent location?
Flags: needinfo?(wlachance)
(In reply to Dave Hunt (:davehunt) from comment #11)
> Can we close this bug, and raise new bugs for improvements such as moving it
> to a more permanent location?

Makes sense to me.
Flags: needinfo?(wlachance)
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Depends on: 960063
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.