1208427 - 5% Win8 tcanvasmark regression on Mozilla-Inbound (v.44) on September 23, 2015 from push 1a17f2ca929d

Status: RESOLVED WONTFIX

(Core :: Graphics: Layers, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I hestitate to mark this as a regression, as we went from *bi-modal* to consistency, but nevertheless we have a regression based on the fact that we now only report one of the two results, unfortunately the one with lower performance.

Here is a view of the graphs:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-inbound,0779d5388323593e99f9e1810a92950d8e70fb9e,1]&series=[mozilla-inbound,7ee51eca80af0b3b6f2904e78ed0418bb9c7ace2,1]&series=[mozilla-inbound,7f91b9eeed83336139df560492dfb40965f5d041,0]&series=[mozilla-inbound,6c3c1bf987447ac87174f1e3c7e7053b85f4fde8,0]

This affects pgo, opt, e10s, and non-e10s.

I have done some retriggers on e10s opt:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=f721299f9dca&tochange=44004b0d91bd&filter-searchStr=windows%208%20x64%20opt%20talos%20performance%20talos%20chrome&group_state=expanded

and you can see that we had a mix of high/low results on the changesets prior and when bug 1194335 landed we stay in the lower range (~8400).

This is exciting as we have more consistent results- I would really like to figure out why this happened, maybe it can help clue us into other tests which are bi-modal.

:jmuizelaar, can you give some insight as to why canvasmark might react this way on windows 8 to your change?  I assume that we have a specific graphics card for the windows 8 testers only.

Comment 1

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #0)
> we have a regression based on the fact that we
> now only report one of the two results, unfortunately the one with lower
> performance.

I'm not sure I understand who is "we" at "we now only report".

Does it mean that we deployed some "anti-bimodal" filtering which just drops the higher values?

Or that the test itself was having bimodal results but stopped doing that for reasons we don't know?

Comment 2

[Avi Halachmi (:avih)]

Ah. Missed the last part of the message. Apparently the second option ("the test stopped having bimodal results and we still don't know why").

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

s/we have/there is/

Talos graphs have historically had bi-modal data, now it is not bi-modal thanks to the patch to Firefox.  There were no talos changes which would account for this, and retriggers show that old revisions post bi-modal which rules out machine changes.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

This regression is not too surprising. The patch disabled an optimization on Nvidia cards. We're going to have to take it for now because we're doing some exploration into the crashes in bug 1189940 and we suspect this optimization might be to blame.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

This is good to know!  What is interesting is that our bi-modal results are now more uniform, this makes me wonder if the optimization was only being hit part of the time, now it is hit none of the time.  Jeff, does that make any sense?

Regarding this bug, do you think we should close this as wontfix and if the work in bug 1189940 is landed we can enjoy any gains at that time?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

this also affects tscrollx by 5% on windows 8.

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

We're going to uplift this performance regression in bug 1194335, in order to deal with crashes tracked in bug 1189940.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

enough other random changes in the numbers resulted in us not seeing an alert after the uplift for this.  do we plan to work on this, or should we close it?

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

We will close this one.