860516 - Run AWFY benchmarks on B2G.

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Nicolas B. Pierron [:nbp]]

I am opening this Bug to track the status of running AWFY benchmarks on the B2G browser.  At the moment:

- I got one Unagi
- I got my server making B2G builds.
    B2G builds are only using m-c builds and not inbound builds, I will probably change that as the same machine already converts all hg commits to git.

- I am able to flash the device from the server.
- I am able to run marionette and gaia-ui-tests.

I will probably fork the gaia-ui-tests to add a benchmark part to it which will run the JS benchmarks inside the B2G browser.

Comment 1

[Nicolas B. Pierron [:nbp]]

Attachment: AWFY: Extract output filters (Link to pull request)

This patch extract the filter phase out of the harness function. The filter phase is then reuse in a standalone phase to produce the data structure expected by the submitter out-of a shell-like output.

That way, running benchmarks on B2G and providing the same shell-like output makes things easier for humans to check, and for tools to process (without adding yet another input format).

Comment 2

[Nicolas B. Pierron [:nbp]]

Attachment: Add browser benchmark to gaia-ui-tests (link to pull request)

These patches are adding marionette instrumentation to run octane benchmarks in B2G's browser application to be used for AWFY.

The added test, go to one octane web-site, then let the phone run the benchmark and collect the result and provide a shell-like output of the benchmark to make it compatible with Are We Fast Yet filters.

Comment 3

[Nicolas B. Pierron [:nbp]]

Comment on attachment 736975 [details]
Add browser benchmark to gaia-ui-tests (link to pull request)

This change has been refused because the repository that I forked is dedicated to QA, and apparently performance tests do not belong to this repository.

Another the repository [1] is holding performance tests which are reported on tbpl, which does not seems to be the right place as we don't want to duplicate the number of times we run these benchmarks and as the JS Team mostly focus on what Are We Fast Yet reports.

So in the mean time the changes will remain in the "bench" branch of the fork I made of "gaia-ui-tests" [2].

[1] https://github.com/mozilla/b2gperf
[2] https://github.com/nbp/gaia-ui-tests/tree/bench

Comment 4

[Nicolas B. Pierron [:nbp]]

For the notice,
- The Unagi is benchmarking well, still has a 1-2% noise.
- The results are available on awfy [1], Thanks dvander for the awfy support :)
- I have a blog article to be reviewed on the JavaScript blog of mozilla.
- I am trying to tackle some wifi issues.  Soon everybody would irradiated at 2.4 GHz!

[1] http://arewefastyet.com/#machine=14

Comment 5

[Nicolas B. Pierron [:nbp]]

- A blog post is being review on this topic and I will close this bug as soon as the blog post is published.

Comment 6

[Nicolas B. Pierron [:nbp]]

(In reply to Nicolas B. Pierron [:nbp] from comment #5)
> - A blog post is being review on this topic and I will close this bug as
> soon as the blog post is published.

I will mark this bug as fixed, because apparently it is impossible to get a device which can be used for dev & benchmarks which is not a Unagi and with which we are allowed to write about publicly.

It appears that we are not even supposed to use the name Unagi publicly (so not even on bugzilla … ?), even if this is a code name … what the ?!