1006531 - Enhance talos to test OMTC performance

Status: RESOLVED DUPLICATE of bug 1189901

(Testing :: Talos, defect)

Description

[Avi Halachmi (:avih)]

Currently talos' tscroll and tsvgx iterate frames using requestAnimationFrame (in ASAP mode) and then report average iteration duration.

On cases where the bottleneck is main thread performance, it means that the composition thread's throughput is not reached and therefore not measured. (regardless, composition thread's iteration intervals are currently not measures, though it could be added).

On an IRC chat with kats and mattw, we concluded that we could utilize APZC scroll-trigger API (bug 928839) to fire an OMTC only scroll which would not be limited by main thread, and the composition thread's intervals would be a reliable gauge for compositions throughput.

This bug is about modifying tscroll to also use this API, and then take both results (rAF and APZC iterations) into account.

I'm not sure yet if tsvgx should also be modified.

Comment 1

[Avi Halachmi (:avih)]

(In reply to Avi Halachmi (:avih) from comment #0)
> On an IRC chat with kats and mattw, we concluded that we could utilize APZC
> scroll-trigger API (bug 928839) to fire an OMTC only scroll which would not
> be limited by main thread, and the composition thread's intervals would be a
> reliable gauge for compositions throughput.

Though we'll have to make sure that we measure composition intervals rather than APZC iterations intervals. The latter can be great 60Hz while the former could be slower, resulting in checkerboarding.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

The IRC conversation referenced in comment 0 can be found starting here: http://logs.glob.uno/?c=mozilla%23developers&s=31+Mar+2014&e=1+Apr+2014#c921441

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Avi, I read through the discussion in comment 2 again and was talking with Vlad and Jeff about this. While it certainly makes sense to use APZC scrolling to test performance once APZC is actually enabled, it doesn't really make sense to test OMTC (without APZC) perf using APZC scrolling.

I can put together (or re-purpose) an API to trigger APZC scrolling, but it would only work on B2G currently (or Metro, which we don't care about). Presumably you can write your test against that and test it on B2G and then later when we enable APZC on other platforms we can run the test on those platforms too.

Is that acceptable? If so, let me know where you plan on invoking this API from (chrome JS, probably?). I was originally thinking of waiting for the smooth-scroll spec to stabilize but it's probably not worth waiting for that, and just add something for our own use right now, modifying it later as we need to. Presumably you'd want to just send a message that says, in effect, "do a smooth scroll to coordinates x,y" and on the compositor side we'd kick off an animation that does that. If this is the case, what metrics are you going to be measuring to test the performance? That part wasn't very clear to me.

Comment 4

[Avi Halachmi (:avih)]

The way I see it, we all have the same common goal, which is to be able to measure our OMTC performance and detect related regressions reliably (probably scroll only at first).

If you think this is not important enough or desirable, then we should sync on this, but I believe we're on the same page.

So this would work best IMO if you guys provide the OMTC expertise, I provide the talos/testing knowledge, and together we come up with something which we're all happy with.

As far as I'm concerned, I don't have "acceptable" or "unacceptable" terms. I certainly wouldn't want you to go out of your way just to "satisfy" me with something which you'd consider premature.

If currently it doesn't look like the APZC API would be a good approach for this, then we'll just need to find another one, or wait until this API or another approach becomes usable. You're the experts, your call how to trigger the test and you know better than anyone when we're ready to use this code and start testing OMTC.


(In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)
> Presumably you'd want to just send a message that says, in
> effect, "do a smooth scroll to coordinates x,y" and on the compositor side
> we'd kick off an animation that does that.

True.

> If this is the case, what metrics are you going to be measuring
> to test the performance? That part wasn't very clear to me.

I'm open to any approach you'd think would give us good (=representing our performance) results and be consistent and reliable. The metric I've been using so far is "average ms per iteration" where lower is better. I don't have ideas for better metrics right now, but if you got any, shoot.

It probably has to have one important characteristic: it shouldn't let the compositor rest, so the result[s] would represent the maximum throughput per test case (or else all the results show 16.7ms and we can't detect changes). Our current scroll tests and few others as well use "ASAP mode" which if effectively un-throttled main thread iteration rate, and (on OS X) non-blocking swap.

We could probably also flood the compositor without ASAP mode or equivalent, by making it compose stuff which we know for certain will take more than 16.7ms per composition. But since this is very dependent on many factors, and it probably wouldn't work for "easy" cases which we'd also like to measure, this approach is probably not the best, and ASAP mode or equivalent would have to be used.

As for how we're going to grab the compositor iterations rate, there's an API which mstange updated for OMTC (bug 920123): Start[/Stop]FrameTimeRecording. So this should work as well.

As for the test cases which we're going to scroll, this would probably be tscroll[x]'s set of synthetic pages, and probably also the tp5 corpus or a subset of it (bug 1006551).

What's next? :)

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Ok, that all makes sense.

I think perhaps the best thing to do for now is to reuse an existing API we have to trigger APZ-based scroll animations, and use that to write the sort of test you describe above. The API I had in mind was the zoom-to-rect API which is already plumbed in and being used by the B2G browser [1]. The zoom-to-rect animation is defined in the APZ code to take 0.25 seconds, so if you do a bunch of those in quick succession you can keep the compositor busy for as long as you need.

Let's say for example we're in ASAP mode and the composites are taking 5ms each. The animation will still take 0.25 seconds, so you'll have 50 composites in the animation. If the composites start taking longer, the number of composites in the animation will drop. So measuring the number of frames composited might be a good metric for this sort of test. I don't know if you have the ability to do that right now.

[1] http://mxr.mozilla.org/mozilla-central/source/dom/browser-element/BrowserElementPanning.js?rev=31631e0d4ada#553

Comment 6

[Avi Halachmi (:avih)]

OK, some alignment of goals and slight change of plans.

What I had in mind for testing OMTC was mostly desktop oriented. The reason being that we're about to ship OMTC on windows/linux soon/now, and we want something to help us detect regressions.

While many of the current tests are affected by OMTC, sometimes even much, they don't try to test OMTC explicitly. But we still got some useful coverage of OMTC already.

So, after talking to kats on irc, it became clear that the suggestion on comment 5, while solid, would not be possible on desktop builds for probably at least several months (kats' estimation of when APZC will come to desktop platforms), and this is too late for the original goal I had in mind for this bug.

kats then suggested on irc that maybe we could test using OMT animations (probably with ASAP), and that jrmuizel might be able to help. He's on cc now.

I also got an email from benwa with a suggestion which apparently vladan asked him to do, though we haven't discussed it yet so neither him nor me are yet sure of its relevance. Here's benwa's suggestion:

---------------------------
IMO the best way to do that is, ideally with static content, to use
the following preferences:

layers.offmainthreadcomposition.frame-rate;0

This will recomposite the same layer tree as fast as possible. We
should be able to hit 60 FPS with this mode on all page that have a
reasonable layer tree.

You can also change the layout framerate to uncapped which will also
uncap the compositor and let it run above FPS for a TART like
compositor test.
---------------------------

I think we're now back to square 1: decide what/how to test, and the approaches suggested in this comment don't include scroll, which isn't a problem. We can create a new test for this and it doesn't have to piggyback tscroll.

Our goal here is to create a guage which the gfx guys would believe when it says that some aspect of OMTC improved or regressed.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

:avih, is this still a valid concern?  I am not sure if we have replace OMTC with anything

Comment 8

[Avi Halachmi (:avih)]

kats should be the ultimate judge here, but my view is as follows:

1. AFAIK we still don't have a stress test for the compositor, and IMO it could still be useful to be able to track compositor regressions without being limited by main thread layout delays. But I don't have a useful approach in mind for that.

2. Bug 1189901 is about testing APZ scrolling, and undoubtedly there's some overlap between "testing the compositor" and "testing APZ scroll". While I hope it would stress the compositor in some way, it's limited to running with e10s only (since APZ works only in e10s), _and_ I'd think it would also represent main thread layout delays (checkerboarding - which happens when the main thread can't layout fast enough).

Whether or not we could have some APZ sub-test to reflect compositor-only is not something I know.

Overall, we can probably make this bug a dup of bug 1189901, unless kats thinks otherwise.

kats?

Comment 9

[Kartikaya Gupta (email:[e-mail])]

I agree with comment 8. I think any APZ talos test will also exercise the compositor, and vice-versa (assuming APZ is enabled) so I think we should just write one test for bug 1189901 which will pick up both regressions in APZ and the compositor. I'm happy to dup this bug over.

Comment 10

[Avi Halachmi (:avih)]

Thanks.