747515 - Add a self-serve web interface for running Eideticker tests against a particular build

Status: RESOLVED WONTFIX

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

It would be useful if developers could run tests and analyze results without needing to ssh into the machine and invoke bizarre command-line incantations. Ideal would be a web-based from that they could use to submit an apk and a set of tests that they'd like to run, with results both emailed and available in a nice-looking web ui.

Comment 1

[Benoit Girard (:BenWa)]

Related to bug 770144 and wanting to run with custom build. How hard would it be to add a temporary privileged solution to run a custom build, optional change the default prefs and get a result out with a profile+video? Something over SSH for a few trusted individuals is fine.

Comment 2

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #1)
> Related to bug 770144 and wanting to run with custom build. How hard would
> it be to add a temporary privileged solution to run a custom build, optional
> change the default prefs and get a result out with a profile+video?
> Something over SSH for a few trusted individuals is fine.

Not hard, maybe an (undisturbed) day or two of effort. Probably need to:

(1) Refactor the profiling code to not be tied to the dashboard
(2) Make the graph view of results (http://people.mozilla.com/~wlachance/cnn-benchmark/metric.html?data=metric-test-1343419943.79.json#/checkerboard) be more intuitive to create
(3) Add profiling information to the graph view of results
(4) Add option to change preferences

Comment 3

[William Lachance (:wlach)]

BenWa pointed out that just doing (1) and (4) would probably be enough to help him and :Cwiiis  with their updates to incremental tiled rendering.

Patch addressing (1): https://github.com/mozilla/eideticker/commit/66b51a08f1a408efb9d4f8b6c0857cac3a039ed3

I'll add something for (4) shortly.

Comment 4

[William Lachance (:wlach)]

Ok you can now specify command line arguments like this:

runtest.py --no-capture --extra-prefs "{\"layers.acceleration.frame-counter\": true }" org.mozilla.fennec src/tests/ep1/taskjs.org/index.html

Comment 5

[Scott Johnson (:jwir3)]

(In reply to William Lachance (:wlach) from comment #4)
> Ok you can now specify command line arguments like this:
> 
> runtest.py --no-capture --extra-prefs
> "{\"layers.acceleration.frame-counter\": true }" org.mozilla.fennec
> src/tests/ep1/taskjs.org/index.html

Nice!  Thank you!!

Comment 6

[William Lachance (:wlach)]

Dave Hunt has some ideas for getting jenkins to do this.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Jenkins supports parameterised build, so we could create a job that takes the build location URL as a parameter. The job would then download the build from that location and run the tests. We could add e-mail notifications of the results (with attachments), and Jenkins could also serve as the results dashboard. As this bug appears to be related to Firefox for Android, this proposal will depend on bug 1015382. It would be useful to identify a typical command line for one of these adhoc tests.

Comment 8

[BMO Automation]

Eideticker has been discontinued, see bug 1361056