Closed Bug 990842 Opened 7 years ago Closed 6 years ago

Start Running Graphics App On Datazilla to test Graphics Performance

Categories

(Firefox OS Graveyard :: Performance, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED WONTFIX
1.4 S6 (25apr)

People

(Reporter: mchang, Assigned: mchang)

References

Details

(Keywords: ateam-b2g-device-task, perf, Whiteboard: [c=automation p= s=2014.08.15 u=])

Start running some of the micro benchmarks from the gfx app (https://github.com/vvuk/gfxapp) on datazilla on a flame device. This will enable us to see how the graphics component itself is performing.
What's the best way to integrate this onto datazilla? What devices are currently on datazilla? Thanks!
Flags: needinfo?(jgriffin)
Is this just the 7 .html files that need to be loaded?  that would be 7 tests where each test loads a different webpage.  How are the results reported?

I know for talos and b2gperf we use a python client to upload data to datazilla, I am not sure if there is a toolchain or example using javascript to upload results, right now in those examples I don't see anything writing results to a file or a server.

I was going to look at what devices were on b2gperf, but the jenkins machine is not accessible.

One thing with datazilla is you need to post to a project.  Will this be run with b2gperf or another toolchain?  If not, then we need to add a new project to datazilla.  quite possibly we could use the same UI as we do for b2gperf.
(In reply to Joel Maher (:jmaher) from comment #2)
> I was going to look at what devices were on b2gperf, but the jenkins machine
> is not accessible.

Jenkins is available, but since the MV move it's now at http://selenium.qa.mtv2.mozilla.com:8080/

We currently run the b2gperf tests on hamachi and inari devices.

> One thing with datazilla is you need to post to a project.  Will this be run
> with b2gperf or another toolchain?  If not, then we need to add a new
> project to datazilla.  quite possibly we could use the same UI as we do for
> b2gperf.

We could most likely use the same project as b2gperf (b2g), even if we're not using that tool.
do we have an idea of what harness will run these tests?

davehunt, on the mozillaVPN, I am unable to access: selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access permisions needed.
(In reply to Joel Maher (:jmaher) from comment #4)
> do we have an idea of what harness will run these tests?
> 
> davehunt, on the mozillaVPN, I am unable to access:
> selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access
> permisions needed.

Stephen: Can we get Joel access? Should we need to request access for individuals?
Flags: needinfo?(stephen.donner)
We'll need a harness that wraps this app, assuming that it isn't bundled into a normal B2G build.  We can probably write a gaia-ui-test for this, to install the app and interact with it.

To report results to datazilla, we'll need the tests to output JSON to a result file.  We can probably build this into "wrapper" gaia-ui-test.

Dave, does this approach make sense?
Flags: needinfo?(jgriffin)
(In reply to Dave Hunt (:davehunt) from comment #5)
> (In reply to Joel Maher (:jmaher) from comment #4)
> > do we have an idea of what harness will run these tests?
> > 
> > davehunt, on the mozillaVPN, I am unable to access:
> > selenium.qa.mtv2.mozilla.com:8080, quite possibly there is some other access
> > permisions needed.
> 
> Stephen: Can we get Joel access? Should we need to request access for
> individuals?

Yes, unfortunately right now, the way to get folks access to boxes in the QA lab in Mountain View (which is on a separate VLAN) is to get a SecOps bug filed, requesting access :-(
Flags: needinfo?(stephen.donner)
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> We'll need a harness that wraps this app, assuming that it isn't bundled
> into a normal B2G build.  We can probably write a gaia-ui-test for this, to
> install the app and interact with it.
> 
> To report results to datazilla, we'll need the tests to output JSON to a
> result file.  We can probably build this into "wrapper" gaia-ui-test.
> 
> Dave, does this approach make sense?

It sounds reasonable to me.
Is Core/Graphics the right component for this feature?
If the only thing that's needed is the wrapper harness and reporting to datazilla, then we can move it to Firefox OS:General.
Component: Graphics → Performance
Product: Core → Firefox OS
Assignee: nobody → mchang
Summary: Starting Running Graphics App On Datazilla to test Graphics Performance → Start Running Graphics App On Datazilla to test Graphics Performance
Target Milestone: --- → 1.4 S6 (25apr)
Whiteboard: [c=automation p= s= u=] → [c=automation p=3 s= u=]
Ok, so if I understand correctly, we have a few issues.

1) We have to change the tests to output data in JSON. I don't think we have to run all the gfx app tests, all them other than the 25x25 tiling ones. I think what we want to measure is just the FPS.
2) Do we have a nice clean way to remove data from logcat? e.g., if we dump things to log cat, one for each .html file, is it easy to separate them? Would it just be easier to make 7 separate apps, one for each file?
If the app could write results to a file on the phone (in some predictable location) that we could pull using adb, that would probably be easiest.  We could also potentially grep it out of the logcat.  We don't really have a "nice clean way" to do this, but it's possible.  As long as the data we're writing allows the html files to be distinguished, it doesn't really matter if it's one app or seven.
Resolving due to lack of time to do this. Reopen if need be.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Whiteboard: [c=automation p=3 s= u=] → [c=automation p= s=2014.08.15 u=]
You need to log in before you can comment on or make changes to this bug.