968290 - [meta] Report Layer Overdrawing Numbers in Make test-perf

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia, defect, P2)

Description

[Mason Chang [Inactive] [:mchang]]

We should report the overpainting numbers into Travis (the 3rd number on the FPS counter). It should help us catch extreme overpainting cases.

Comment 1

[Zac C (:zac)]

This sounds like something for b2gperf/datazilla rather than the functional tests?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Are these overpainting numbers available via API or logcat? We could add this as a new metric in the b2gperf scroll tests, or we could potentially retrieve these metrics for each functional test and report to the console/log. If we knew a sensible threshold we could even warn or fail the test as a result.

Comment 3

[Mason Chang [Inactive] [:mchang]]

I could quickly write a patch that makes it available in logcat or an API. What would work best for b2gperf scroll tests? At the moment, it's just drawn onto the screen when you enable 'Show FPS Counters'.

Comment 4

[Benoit Girard (:BenWa)]

Getting overdraw number is useful. Keep in mind that we want to accept regressions here if we're requesting active layers that were not there previously. Tracking this to watch out that we're not regressing this unintentionally is good.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Mason Chang [:mchang] from comment #3)
> I could quickly write a patch that makes it available in logcat or an API.
> What would work best for b2gperf scroll tests? At the moment, it's just
> drawn onto the screen when you enable 'Show FPS Counters'.

So at the moment b2gperf gathers the FPS numbers from the logcat as seen here: https://github.com/mozilla/b2gperf/blob/5d9d7a786e3a3f7b49d3fea9fc66caad85ceac66/b2gperf/b2gperf.py#L496

If there's another metric to extract from the logcat it would be fairly easy to add it to b2gperf, however I believe the ultimate goal is to wind down b2gperf in favour of the make test-perf in gaia. Also, the scrolling tests in b2gperf are currently blocked by bug 958036.

Comment 6

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(In reply to Zac C (:zac) from comment #1)
> This sounds like something for b2gperf/datazilla rather than the functional
> tests?

We discussed this briefly yesterday and there did seem to be some desire for a yes/no sanity test built around this in Travis. Mason seemed confident this would be a quick-running test and that finding a threshold for "gross regression" such that you can do this is possible. Apparently, the results should also be a little more deterministic than FPS-style stats, so should be repeatable.

I do agree that generally this is something more for a perf harness, but if all of the above are true and I haven't misunderstood, I guess standing up a sanity test makes some amount of sense.

Comment 7

[cmtalbert]

Having the data available via APIs is generally better than logcat unless the API call changes what you're measuring. In that case, logcat works. But either way, we should wire this kind of information everywhere we see it having an actionable benefit.

Comment 8

[Zac C (:zac)]

Move to general because it's not pertaining to the UI Tests.

Comment 9

[Mason Chang [Inactive] [:mchang]]

We can't have these numbers in travis because travis runs on desktop-b2g which has very different overfill numbers compared to a device. These numbers have to run on device and therefore make test-perf. Then we can run these numbers on datazilla.

Comment 10

[Benoit Girard (:BenWa)]

We've had several overdraw regression. Mostly recently bug 1052095 and a host of similar bug as denoted by the regression window. Getting a test for this is useful for both performance and power usage. Let's try to up this in the backlog when we have time for it.

Bug 1052095 includes one testcase that should be included.