1013262 - 8-37% regressions on many tests due to OMTC on May 20 (Fx32) from inbound revision 718a9852b60d

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

As expected there are some tests showing *regressions* from OMTC.  I am glad we have this turned on and look forward to accepting some changes in talos tests, while maybe fixing some of this.  Here is the breakdown of the changes:

Win8:
* TResize 37.6% regression: http://mzl.la/1jxq3KL
* SVG-ASAP 12.2% regression: http://mzl.la/1jxpPDs
* TART 22% regression: http://mzl.la/1tcCkWM
* CanvasMark 13% regression: http://mzl.la/1tcCjlN
* CART 17.1% regression: http://mzl.la/1tcCeyk

Win XP:
* TResize 24% regression: http://mzl.la/1nfzDQy
* tp5o responsiveness 8.47 regression: http://mzl.la/1nfzqwL


And now for the improvements:
* Win8 - tscroll-asap, 13.7% improvement: http://mzl.la/1tcCmhp
* WinXP - SVG-ASAP, 17% improvement: http://mzl.la/1nfCMjk

Please speak up for which results come as a surprise.

Comment 1

[Bas Schouten (:bas.schouten)]

Thanks for documenting these. The Tresize on Win8 surprises me a little bit. It's something I'll look into first once I get past the remaining correctness bugs.

Comment 2

[Avi Halachmi (:avih)]

FWIW, tresize does _not_ use ASAP mode so at least here we can know it's unaffected by its potential quirks. Yet many times it detects perf changes which correlate with those which TART detects, as apparently it exercises similar gfx "pain" paths as TART does.

Apparently tresize doesn't need ASAP because each of its iterations is slow enough to take more than 16.7ms, which in practice ends up with reliable results.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Also:
Win7:
* Canvasmark: 7.53% regression: http://mzl.la/1qXk4Db

Comment 4

[Matt Woodrow (:mattwoodrow)]

Bug 1013759 fixed a regression with OMTC, and looks like it's making a decent dent into the TART and tsvgx regressions.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, tsvgx is about half way back to where it started!

tart and canvasmark saw improvements, but only slightly.  Progress!

Comment 6

[Avi Halachmi (:avih)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #4)
> Bug 1013759 fixed a regression with OMTC, and looks like it's making a
> decent dent into the TART and tsvgx regressions.

I'm glad to see that you're able to find real causes for the reported regressions, and that genuine improvements at the code make the numbers go down.

My biggest concern these days is that the tests are less reliable than before because they are maybe more biased with OMTC. It's very hard to evaluate this concern objectively and independently, but as long as you can make real fixes which move the numbers in the expected direction, it supports the position that the test are still useful.

Thanks.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Ok, another regression now that PGO builds are out:
Win7:
* tp5o: 4.4% regression: http://mzl.la/1jIecJV

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

and more pgo builds:
Win8 pgo:
* svgx 15% regression: http://mzl.la/1jI9TP0
* tart 27.8% regression: http://mzl.la/1jI9Oe7
* canvasmark 15.6% regression: http://mzl.la/1jI9JXD

winxp pgo:
* session restore 2.48% regression: http://mzl.la/1jI9A6z
* session restore no auto restore, 2.22% regression: http://mzl.la/1jIdwEp
* tp5o responsiveness 10.4% regression: http://mzl.la/1jI9Y56

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

these have hit aurora now, Bas, can you comment on which regressions listed above are expected?

Comment 10

[Bas Schouten (:bas.schouten)]

I don't think I have any additional information here. We've added several fixed for 'real' performance issues. Some of those improve the scores somewhat as well, the important ones we're uplifting to Aurora. But for now we haven't really had any user complaints about performance that I know of.

Comment 11

[Avi Halachmi (:avih)]

(In reply to Bas Schouten (:bas.schouten) from comment #10)
> But for now
> we haven't really had any user complaints about performance that I know of.

Unless there's a channel which we're listening to, and where general user complaints, performance included, are posted to and it has a track record of reflecting users concerns reliably, I don't think we can expect to hear any such complaint without actively trying to find them ourselves, which we don't do AFAIK.

OTOH, we do know that OMTC improve some things (such as 30% improvements with scrolling) while hurting others, sometimes considerably (like ~100% regression on tab animations). I'm not actively following all test suites, so I'm not sure how many regressed or improved by OMTC.

What to do with that info is another question, but I don't think the answer to this question should depend on users complaints which we've "heard" - when we're not actively trying to listen to such complaints.

It's definitely not an easy question to answer though (what to do about those regressions), but it has to be answered, and if we don't actively decide, then we're implicitly saying "they're acceptable".

I'd prefer to hear such statement explicitly rather than implicitly.

Milan, what do you think?

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

It would be nice to say that "no news is good news", but that probably won't fly in this case.

I would like to track these numbers, possibly in this bug, but until we decide to leave OMTC on in the beta/release, we would probably just look for easy wins and wait until tiling is done on the desktop and then optimize things once.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

so we just enabled OMTC on beta again, I haven't seen any work to reduce these performance regressions.  There are improvements though, maybe we can come to a resolution on this bug?

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

Agreed, let's sort this out - do we have fresh measurements, as in, what the current numbers look like? I think we can just close this; Bas, are we tracking some of the performance issues outside of the tests?

Comment 16

[Bas Schouten (:bas.schouten)]

(In reply to Milan Sreckovic [:milan] from comment #15)
> Agreed, let's sort this out - do we have fresh measurements, as in, what the
> current numbers look like? I think we can just close this; Bas, are we
> tracking some of the performance issues outside of the tests?

We have implemented a number of orthogonal perf improvements since then (which are riding the trains), we were unable to identify any specific causes for the regressions caused by OMTC (the improvements are a little easier to rationalize), so we've assumed they're mostly due to the additional bandwidth required for double buffering. I have recently discovered something which might be indicative of a perf bug with OMTC though, but I haven't been able to follow-up on it yet.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

I vote for closing this, we are shipping with OMTC soon and it sounds like we have done our due diligence to find optimizations where possible.  Please let me know if there is anything I can get or help test.