1060460 - 5.68% tart windows regression on fx team (v.34) August 15 from bug 862563

Status: RESOLVED FIXED

(Firefox Health Report Graveyard :: Client: Desktop, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

here is a winxp tart graph:
http://graphs.mozilla.org/graph.html#tests=[[293,132,37]]&sel=1406739400056,1409331400056&displayrange=30&datatype=running

I did a bunch of retriggers and the values go up from (5.7-6.0) -> (6.0-6.25):
https://tbpl.mozilla.org/?tree=Fx-Team&startdate=2014-08-15&enddate=2014-08-17&jobname=Windows%20XP%2032-bit%20fx-team%20talos%20svgr

Comment 1

[Georg Fritzsche [:gfritzsche]]

Attachment: Don't show the data submission policy notification for Talos

The only thing that stands out to me is that we don't pref off the notification yet for Talos.

Comment 2

[Avi Halachmi (:avih)]

While I understand (and agree with) the approach to disable first time notice for TART measurements, do we know why it actually regressed? Did anyone look at a talos TART run without this patch, identified when is the message displayed, and noticed that it's the message itself which regresses TART in almost 6% overall? (and this 6% is over 10 different animation repeating 25 times)

Comment 3

[Georg Fritzsche [:gfritzsche]]

I would have liked to confirm this via try pushes, but given that Talos is in a different repo that's not an option.
Bug 862563 didn't really touch UI or animations, the notification is the most obvious candidate and i'd rather try the impact of this out before putting in a lot of time here. Any better suggestions?

Comment 4

[Avi Halachmi (:avih)]

I tried to follow bug 862563, but the patch is pretty big and the list of comments is quite long. Could you please explain what does this bug change, and how it would affects TART?

Comment 5

[Georg Fritzsche [:gfritzsche]]

This is a change about the "accepting data submission" handling.
We now show a notification about this very early and don't require active acceptance.

So, the only part that is expected to affect CART here is the notification, the rest of the changes are reworking things for the acceptance handling.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

this is a valid patch and will probably help keep our numbers more sane-  the discussion in this bug could change that, but r+ for now.

Comment 7

[Avi Halachmi (:avih)]

Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

Please don't disable this pref before we understand how it regresses TART so much.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

hmm, there are linux/linux64/win8 tart regressions in the same time frame, I suspect these are related as preliminary hunting continues to point to bug 862563.

Comment 9

[Avi Halachmi (:avih)]

Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

So, I tested it locally (m-c without the talos patch to disable the notification).

It seems that about a minute after the browser starts (around the 4th/5th TART replicate), the notification shows and stays there until the end of the run. It doesn't flicker etc, so not sure what could cause the regression.

Visually I can't say I noticed worse perf once the notification is displayed, but that's expected - though it still doesn't mean that there's no regression or that it wouldn't be visible on slower systems.

Here's the summary (I ran it for 16 replicates and the notification shows at about the 4th - that's on each of the "values" lines from the 4th/5th value onwards):

> ------- Summary: start -------
> Number of tests: 30
> 
> [#0] simple-open-DPI1.half.TART  Cycles:15  Average:4.27  Median:4.23  stddev:0.21 (5.0%)  stddev-sans-first:0.20
> Values: 4.0  4.2  4.3  4.1  4.9  4.2  4.3  4.3  4.2  4.2  4.3  4.2  4.1  4.3  4.3
> 
> [#1] simple-open-DPI1.all.TART  Cycles:15  Average:5.21  Median:5.20  stddev:0.23 (4.3%)  stddev-sans-first:0.22
> Values: 4.9  5.1  5.2  5.0  5.9  5.0  5.3  5.2  5.1  5.3  5.3  5.1  5.0  5.3  5.3
> 
> [#2] simple-open-DPI1.error.TART  Cycles:15  Average:33.22  Median:32.99  stddev:2.04 (6.2%)  stddev-sans-first:2.12
> Values: 33.0  31.9  31.4  33.0  36.3  31.8  33.1  34.0  31.9  38.3  33.2  32.2  33.5  30.0  34.7
> 
> [#3] simple-close-DPI1.half.TART  Cycles:15  Average:5.76  Median:5.83  stddev:0.21 (3.5%)  stddev-sans-first:0.18
> Values: 5.3  5.5  5.5  5.6  6.1  5.8  5.7  5.8  5.8  5.8  5.8  5.9  5.8  5.8  6.0
> 
> [#4] simple-close-DPI1.all.TART  Cycles:15  Average:6.41  Median:6.39  stddev:0.22 (3.5%)  stddev-sans-first:0.20
> Values: 6.0  6.2  6.1  6.3  6.7  6.4  6.3  6.3  6.7  6.7  6.5  6.4  6.5  6.7  6.5
> 
> [#5] simple-close-DPI1.error.TART  Cycles:15  Average:6.84  Median:6.59  stddev:1.31 (19.9%)  stddev-sans-first:1.35
> Values: 7.2  7.9  4.8  6.3  5.4  8.2  5.4  7.6  6.0  7.2  6.4  9.2  6.6  8.7  5.6
> 
> [#6] icon-open-DPI1.half.TART  Cycles:15  Average:4.65  Median:4.61  stddev:0.14 (3.1%)  stddev-sans-first:0.15
> Values: 4.6  4.6  4.7  4.6  5.1  4.6  4.7  4.6  4.6  4.7  4.5  4.7  4.6  4.6  4.6
> 
> [#7] icon-open-DPI1.all.TART  Cycles:15  Average:6.00  Median:5.94  stddev:0.16 (2.6%)  stddev-sans-first:0.16
> Values: 5.8  5.9  6.1  5.8  6.4  5.9  5.9  6.0  5.9  6.0  6.0  6.3  6.1  5.9  5.9
> 
> [#8] icon-open-DPI1.error.TART  Cycles:15  Average:32.84  Median:32.31  stddev:1.62 (5.0%)  stddev-sans-first:1.63
> Values: 34.3  31.9  32.0  33.0  32.0  32.2  29.9  33.4  31.9  30.8  35.1  34.7  35.8  32.3  33.3
> 
> [#9] icon-close-DPI1.half.TART  Cycles:15  Average:5.65  Median:5.65  stddev:0.16 (2.8%)  stddev-sans-first:0.16
> Values: 5.6  5.4  5.4  5.4  5.6  5.6  5.6  5.7  5.7  5.8  5.9  5.7  5.7  5.8  5.8
> 
> [#10] icon-close-DPI1.all.TART  Cycles:15  Average:6.40  Median:6.34  stddev:0.29 (4.6%)  stddev-sans-first:0.30
> Values: 6.3  6.1  6.1  6.0  6.1  6.3  6.2  6.3  6.5  6.7  7.0  6.4  6.6  6.5  6.7
> 
> [#11] icon-close-DPI1.error.TART  Cycles:15  Average:6.40  Median:5.97  stddev:1.82 (30.4%)  stddev-sans-first:1.88
> Values: 6.9  4.1  3.6  6.6  5.8  6.0  5.1  8.4  8.7  8.3  3.9  5.6  5.9  8.0  9.1
> 
> [#12] icon-open-DPI2.half.TART  Cycles:15  Average:5.57  Median:5.67  stddev:0.21 (3.7%)  stddev-sans-first:0.22
> Values: 5.5  5.2  5.1  5.3  5.6  5.7  5.7  5.6  5.7  5.7  5.8  5.6  5.7  5.7  5.7
> 
> [#13] icon-open-DPI2.all.TART  Cycles:15  Average:7.13  Median:7.29  stddev:0.28 (3.8%)  stddev-sans-first:0.28
> Values: 6.8  6.9  6.4  6.8  7.0  7.3  7.3  7.2  7.4  7.3  7.3  7.3  7.3  7.1  7.3
> 
> [#14] icon-open-DPI2.error.TART  Cycles:15  Average:36.56  Median:36.29  stddev:1.77 (4.9%)  stddev-sans-first:1.74
> Values: 38.5  39.8  34.4  37.9  37.4  35.2  36.3  39.2  37.4  36.5  34.6  36.0  34.6  36.1  34.5
> 
> [#15] icon-close-DPI2.half.TART  Cycles:15  Average:5.82  Median:5.94  stddev:0.29 (4.9%)  stddev-sans-first:0.28
> Values: 5.4  5.4  5.2  5.5  5.8  5.9  5.9  6.0  6.0  6.1  6.1  6.1  6.0  5.9  6.1
> 
> [#16] icon-close-DPI2.all.TART  Cycles:15  Average:6.58  Median:6.71  stddev:0.44 (6.5%)  stddev-sans-first:0.43
> Values: 6.1  5.9  5.9  6.1  6.4  6.5  6.5  6.9  6.9  6.9  7.3  6.7  7.1  6.7  6.8
> 
> [#17] icon-close-DPI2.error.TART  Cycles:15  Average:7.57  Median:7.31  stddev:2.55 (34.9%)  stddev-sans-first:2.55
> Values: 5.1  14.6  7.3  4.1  5.4  8.2  7.1  8.6  8.2  6.8  7.5  8.4  9.2  8.9  4.3
> 
> [#18] iconFade-close-DPI2.half.TART  Cycles:15  Average:5.85  Median:5.95  stddev:0.33 (5.5%)  stddev-sans-first:0.30
> Values: 5.3  5.5  5.3  5.4  5.8  5.9  5.9  5.9  6.0  6.1  6.1  6.3  6.2  6.0  6.1
> 
> [#19] iconFade-close-DPI2.all.TART  Cycles:15  Average:5.87  Median:5.98  stddev:0.30 (4.9%)  stddev-sans-first:0.27
> Values: 5.4  5.5  5.4  5.4  5.8  6.0  5.9  5.9  6.1  6.1  6.1  6.2  6.2  6.0  6.1
> 
> [#20] iconFade-close-DPI2.error.TART  Cycles:15  Average:2.52  Median:1.64  stddev:1.73 (105.8%)  stddev-sans-first:1.75
> Values: 1.1  1.1  4.1  1.6  3.4  3.2  5.4  0.4  4.7  0.7  1.0  1.0  3.8  4.7  1.5
> 
> [#21] iconFade-open-DPI2.half.TART  Cycles:15  Average:5.60  Median:5.66  stddev:0.29 (5.2%)  stddev-sans-first:0.28
> Values: 5.2  5.2  5.3  5.2  5.6  5.7  5.7  6.3  5.7  5.7  5.6  5.8  5.7  5.8  5.7
> 
> [#22] iconFade-open-DPI2.all.TART  Cycles:15  Average:5.91  Median:6.00  stddev:0.34 (5.7%)  stddev-sans-first:0.32
> Values: 5.4  5.5  5.5  5.4  6.1  5.9  6.0  6.5  5.9  6.0  6.0  6.2  6.1  6.3  6.0
> 
> [#23] iconFade-open-DPI2.error.TART  Cycles:15  Average:2.91  Median:2.95  stddev:1.53 (52.1%)  stddev-sans-first:1.52
> Values: 1.2  0.3  1.8  1.0  1.8  4.9  4.0  4.4  2.2  2.9  3.7  4.3  1.9  4.8  4.3
> 
> [#24] newtab-open-preload-no.half.TART  Cycles:15  Average:8.12  Median:7.68  stddev:1.18 (15.4%)  stddev-sans-first:0.52
> Values: 12.0  7.7  7.3  7.9  7.8  7.7  7.6  8.4  9.2  8.3  8.1  7.4  7.4  7.3  7.7
> 
> [#25] newtab-open-preload-no.all.TART  Cycles:15  Average:9.68  Median:9.15  stddev:1.22 (13.3%)  stddev-sans-first:0.40
> Values: 13.8  9.0  9.2  9.3  9.5  9.5  9.4  9.7  10.3  9.7  9.6  9.0  9.1  8.8  9.0
> 
> [#26] newtab-open-preload-no.error.TART  Cycles:15  Average:35.07  Median:34.81  stddev:2.54 (7.3%)  stddev-sans-first:2.63
> Values: 34.8  34.3  38.1  32.8  39.5  38.9  36.1  33.9  31.1  35.4  31.3  33.5  36.1  36.7  33.7
> 
> [#27] newtab-open-preload-yes.half.TART  Cycles:15  Average:4.72  Median:4.70  stddev:0.13 (2.9%)  stddev-sans-first:0.14
> Values: 4.6  4.7  5.0  4.5  4.6  4.9  4.7  4.9  4.7  4.8  4.6  4.7  4.7  4.7  4.6
> 
> [#28] newtab-open-preload-yes.all.TART  Cycles:15  Average:6.49  Median:6.42  stddev:0.19 (2.9%)  stddev-sans-first:0.17
> Values: 6.8  6.3  6.8  6.4  6.2  6.3  6.4  6.4  6.7  6.6  6.4  6.5  6.4  6.7  6.3
> 
> [#29] newtab-open-preload-yes.error.TART  Cycles:15  Average:65.32  Median:64.43  stddev:4.05 (6.3%)  stddev-sans-first:2.84
> Values: 76.1  68.6  63.9  67.9  64.2  62.9  64.5  57.3  64.2  64.0  66.7  66.0  67.1  64.4  61.8
> -------- Summary: end --------

I'd say that the numbers do show that some of the tests regress slightly around the 4th replicate. I guess somehow it's harder for the browser to animate the tabs while the notification is displayed.

With that in mind, once we noticed that the regression seems to start with the notification, I'd say go ahead, land the talos patch to disable the notification, and let's see that it "fixes" the regression.

gfritzsche, next time I expect you to do this work ;)
Ping Joel or myself if you need any help.

Comment 10

[Georg Fritzsche [:gfritzsche]]

https://hg.mozilla.org/build/talos/rev/849898ecb9be

Comment 11

[Georg Fritzsche [:gfritzsche]]

Ok, thanks for looking into this Avi - with not having too much time to spare and this patch definitely missing on Talos i rather wanted this to land first.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

all that is left is to modify talos.json in the main tree:
http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json

change the revision it references:
http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json#8

to be what you just landed '849898ecb9be'.  As it stands, there are no other changes in the talos repo, so there is no danger in picking up other changes.  this could be safely uplifted to aurora and beta.

Comment 13

[Georg Fritzsche [:gfritzsche]]

Attachment: Change talos revision

Comment 14

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Comment on attachment 8483389 [details] [diff] [review]
Change talos revision

Review of attachment 8483389 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 15

[Georg Fritzsche [:gfritzsche]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7a721388e46b

Comment 16

[Georg Fritzsche [:gfritzsche]]

Aurora is closed right now, comment 15 still needs uplift.

Comment 17

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #15)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/7a721388e46b

Hm, so this strangely seems to have triggered "WEBGL Terrain" regressions:
http://graphs.mozilla.org/graph.html#tests=[[325,131,35]]&sel=1409656834000,1409829634000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,33]]&sel=1409656425000,1409829225000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,31]]&sel=1409658214000,1409831014000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,37]]&sel=1409658219000,1409831019000&displayrange=7&datatype=running

Those seem to look real, right?
I don't really see other possible causes except maybe the healthreporter being instantiated at a different time, so i'll have to check more into this tomorrow - jmaher pointed me to how to actually run tg1 locally.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7a721388e46b

Comment 19

[Georg Fritzsche [:gfritzsche]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8560dad606d5