774278 - Need to write a performance test tracking repaints

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

The test case is that when the  homescreen is moved, we shouldn't repaint the screen too often. We should move the homescreen, and count how many times we repaint the screen and report this number as a performance metric. This test can be a model for other, similar performance tests.

We can probably listen to MOZ_AFTER_PAINT, but I'll have to test it out.

Comment 1

[Andreas Gal :gal]

Once we have an example test for this we should extend this to all major animations.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Mochitest-chrome supports this.  We need to start with a small testcase.

For regression testing of gaia, we should use something less invasive, like eideticker.  These approaches are complementary.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

In more detail, we should use two approaches:

 - for gecko bugs in which we over-invalidate, we should write minimal testcases that use the mochitest-chrome support for testing invalidation.  These are pretty easy to write.

 - for testing user-perceived perf of the homescreen and animations, we should use a black-box system like eideticker.  It's important there that we hit 60fps, it's not important how invalidation behaves unless it prevents us from hitting 60fps.

Comment 4

[Jonathan Griffin (:jgriffin)]

We haven't yet tried running mochitest-chrome on B2G.  I'm not sure if our work for running mochitest-plain will be sufficient, or if it will need some additional work.

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Chris Jones [:cjones] [:warhammer] from comment #3)
> In more detail, we should use two approaches:
> 
>  - for gecko bugs in which we over-invalidate, we should write minimal
> testcases that use the mochitest-chrome support for testing invalidation. 
> These are pretty easy to write.
> 
>  - for testing user-perceived perf of the homescreen and animations, we
> should use a black-box system like eideticker.  It's important there that we
> hit 60fps, it's not important how invalidation behaves unless it prevents us
> from hitting 60fps.

So for the example test Andreas mentioned, do you want to have a marionette test running this in the interim, until we get support for mochitest-chrome/eideticker, or do you want to wait until they are supported?

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Well, I'm not exactly sure what comment 0 and comment 1 are aiming at.  We could repaint 60 times and hit 60fps, or repaint only once but get 1fps.  Similarly for invalidation.

It would be useful to report the compositing fps from gecko, but (i) that's hard to get at --- can live on another thread or in another process --- and (ii) it's also not representative of user-perceived drawn frames.  That's about the closest we can get from within gecko though.

If the homescreen was suffering because of an invalidation bug in gecko, we should make a small test case for that and write a mochitest-chrome test that runs on tinderbox, on all platforms.

My intuition is that time is better spent on getting our power tools working with b2g/marionette, rather than trying to hack something in right now.  Manual testing with the fps counter and paint flashing is pretty effective, but obviously doesn't scale.

Comment 7

[Jonathan Griffin (:jgriffin)]

I'm closing this as obsolete, since for fps measurements we intend to use Eideticker for some tests, and ted's work for others.