990842 - Start Running Graphics App On Datazilla to test Graphics Performance

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

Start running some of the micro benchmarks from the gfx app (https://github.com/vvuk/gfxapp) on datazilla on a flame device. This will enable us to see how the graphics component itself is performing.

Comment 1

[Mason Chang [Inactive] [:mchang]]

What's the best way to integrate this onto datazilla? What devices are currently on datazilla? Thanks!

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Is this just the 7 .html files that need to be loaded?  that would be 7 tests where each test loads a different webpage.  How are the results reported?

I know for talos and b2gperf we use a python client to upload data to datazilla, I am not sure if there is a toolchain or example using javascript to upload results, right now in those examples I don't see anything writing results to a file or a server.

I was going to look at what devices were on b2gperf, but the jenkins machine is not accessible.

One thing with datazilla is you need to post to a project.  Will this be run with b2gperf or another toolchain?  If not, then we need to add a new project to datazilla.  quite possibly we could use the same UI as we do for b2gperf.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher (:jmaher) from comment #2)
> I was going to look at what devices were on b2gperf, but the jenkins machine
> is not accessible.

Jenkins is available, but since the MV move it's now at http://selenium.qa.mtv2.mozilla.com:8080/

We currently run the b2gperf tests on hamachi and inari devices.

> One thing with datazilla is you need to post to a project.  Will this be run
> with b2gperf or another toolchain?  If not, then we need to add a new
> project to datazilla.  quite possibly we could use the same UI as we do for
> b2gperf.

We could most likely use the same project as b2gperf (b2g), even if we're not using that tool.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

do we have an idea of what harness will run these tests?

davehunt, on the mozillaVPN, I am unable to access: selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access permisions needed.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher (:jmaher) from comment #4)
> do we have an idea of what harness will run these tests?
> 
> davehunt, on the mozillaVPN, I am unable to access:
> selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access
> permisions needed.

Stephen: Can we get Joel access? Should we need to request access for individuals?

Comment 6

[Jonathan Griffin (:jgriffin)]

We'll need a harness that wraps this app, assuming that it isn't bundled into a normal B2G build.  We can probably write a gaia-ui-test for this, to install the app and interact with it.

To report results to datazilla, we'll need the tests to output JSON to a result file.  We can probably build this into "wrapper" gaia-ui-test.

Dave, does this approach make sense?

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Dave Hunt (:davehunt) from comment #5)
> (In reply to Joel Maher (:jmaher) from comment #4)
> > do we have an idea of what harness will run these tests?
> > 
> > davehunt, on the mozillaVPN, I am unable to access:
> > selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access
> > permisions needed.
> 
> Stephen: Can we get Joel access? Should we need to request access for
> individuals?

Yes, unfortunately right now, the way to get folks access to boxes in the QA lab in Mountain View (which is on a separate VLAN) is to get a SecOps bug filed, requesting access :-(

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> We'll need a harness that wraps this app, assuming that it isn't bundled
> into a normal B2G build.  We can probably write a gaia-ui-test for this, to
> install the app and interact with it.
> 
> To report results to datazilla, we'll need the tests to output JSON to a
> result file.  We can probably build this into "wrapper" gaia-ui-test.
> 
> Dave, does this approach make sense?

It sounds reasonable to me.

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

Is Core/Graphics the right component for this feature?

Comment 10

[Jonathan Griffin (:jgriffin)]

If the only thing that's needed is the wrapper harness and reporting to datazilla, then we can move it to Firefox OS:General.

Comment 11

[Mason Chang [Inactive] [:mchang]]

Ok, so if I understand correctly, we have a few issues.

1) We have to change the tests to output data in JSON. I don't think we have to run all the gfx app tests, all them other than the 25x25 tiling ones. I think what we want to measure is just the FPS.
2) Do we have a nice clean way to remove data from logcat? e.g., if we dump things to log cat, one for each .html file, is it easy to separate them? Would it just be easier to make 7 separate apps, one for each file?

Comment 12

[Jonathan Griffin (:jgriffin)]

If the app could write results to a file on the phone (in some predictable location) that we could pull using adb, that would probably be easiest.  We could also potentially grep it out of the logcat.  We don't really have a "nice clean way" to do this, but it's possible.  As long as the data we're writing allows the html files to be distinguished, it doesn't really matter if it's one app or seven.

Comment 13

[Mason Chang [Inactive] [:mchang]]

Resolving due to lack of time to do this. Reopen if need be.