Closed Bug 1060460 Opened 5 years ago Closed 5 years ago

5.68% tart windows regression on fx team (v.34) August 15 from bug 862563

Categories

(Firefox Health Report Graveyard :: Client: Desktop, defect)

defect
Not set

Tracking

(firefox33 unaffected, firefox34 fixed, firefox35 fixed)

RESOLVED FIXED
Firefox 35
Tracking Status
firefox33 --- unaffected
firefox34 --- fixed
firefox35 --- fixed

People

(Reporter: jmaher, Assigned: gfritzsche)

References

Details

(Keywords: perf, regression, Whiteboard: [talos_regression])

Attachments

(2 files)

Points: --- → 3
Flags: firefox-backlog+
The only thing that stands out to me is that we don't pref off the notification yet for Talos.
Assignee: nobody → georg.fritzsche
Attachment #8482298 - Flags: review?(jmaher)
Status: NEW → ASSIGNED
Iteration: --- → 35.1
Flags: qe-verify?
Flags: qe-verify? → qe-verify-
While I understand (and agree with) the approach to disable first time notice for TART measurements, do we know why it actually regressed? Did anyone look at a talos TART run without this patch, identified when is the message displayed, and noticed that it's the message itself which regresses TART in almost 6% overall? (and this 6% is over 10 different animation repeating 25 times)
I would have liked to confirm this via try pushes, but given that Talos is in a different repo that's not an option.
Bug 862563 didn't really touch UI or animations, the notification is the most obvious candidate and i'd rather try the impact of this out before putting in a lot of time here. Any better suggestions?
I tried to follow bug 862563, but the patch is pretty big and the list of comments is quite long. Could you please explain what does this bug change, and how it would affects TART?
This is a change about the "accepting data submission" handling.
We now show a notification about this very early and don't require active acceptance.

So, the only part that is expected to affect CART here is the notification, the rest of the changes are reworking things for the acceptance handling.
Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

this is a valid patch and will probably help keep our numbers more sane-  the discussion in this bug could change that, but r+ for now.
Attachment #8482298 - Flags: review?(jmaher) → review+
Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

Please don't disable this pref before we understand how it regresses TART so much.
Attachment #8482298 - Flags: feedback-
hmm, there are linux/linux64/win8 tart regressions in the same time frame, I suspect these are related as preliminary hunting continues to point to bug 862563.
Comment on attachment 8482298 [details] [diff] [review]
Don't show the data submission policy notification for Talos

Review of attachment 8482298 [details] [diff] [review]:
-----------------------------------------------------------------

So, I tested it locally (m-c without the talos patch to disable the notification).

It seems that about a minute after the browser starts (around the 4th/5th TART replicate), the notification shows and stays there until the end of the run. It doesn't flicker etc, so not sure what could cause the regression.

Visually I can't say I noticed worse perf once the notification is displayed, but that's expected - though it still doesn't mean that there's no regression or that it wouldn't be visible on slower systems.

Here's the summary (I ran it for 16 replicates and the notification shows at about the 4th - that's on each of the "values" lines from the 4th/5th value onwards):

> ------- Summary: start -------
> Number of tests: 30
> 
> [#0] simple-open-DPI1.half.TART  Cycles:15  Average:4.27  Median:4.23  stddev:0.21 (5.0%)  stddev-sans-first:0.20
> Values: 4.0  4.2  4.3  4.1  4.9  4.2  4.3  4.3  4.2  4.2  4.3  4.2  4.1  4.3  4.3
> 
> [#1] simple-open-DPI1.all.TART  Cycles:15  Average:5.21  Median:5.20  stddev:0.23 (4.3%)  stddev-sans-first:0.22
> Values: 4.9  5.1  5.2  5.0  5.9  5.0  5.3  5.2  5.1  5.3  5.3  5.1  5.0  5.3  5.3
> 
> [#2] simple-open-DPI1.error.TART  Cycles:15  Average:33.22  Median:32.99  stddev:2.04 (6.2%)  stddev-sans-first:2.12
> Values: 33.0  31.9  31.4  33.0  36.3  31.8  33.1  34.0  31.9  38.3  33.2  32.2  33.5  30.0  34.7
> 
> [#3] simple-close-DPI1.half.TART  Cycles:15  Average:5.76  Median:5.83  stddev:0.21 (3.5%)  stddev-sans-first:0.18
> Values: 5.3  5.5  5.5  5.6  6.1  5.8  5.7  5.8  5.8  5.8  5.8  5.9  5.8  5.8  6.0
> 
> [#4] simple-close-DPI1.all.TART  Cycles:15  Average:6.41  Median:6.39  stddev:0.22 (3.5%)  stddev-sans-first:0.20
> Values: 6.0  6.2  6.1  6.3  6.7  6.4  6.3  6.3  6.7  6.7  6.5  6.4  6.5  6.7  6.5
> 
> [#5] simple-close-DPI1.error.TART  Cycles:15  Average:6.84  Median:6.59  stddev:1.31 (19.9%)  stddev-sans-first:1.35
> Values: 7.2  7.9  4.8  6.3  5.4  8.2  5.4  7.6  6.0  7.2  6.4  9.2  6.6  8.7  5.6
> 
> [#6] icon-open-DPI1.half.TART  Cycles:15  Average:4.65  Median:4.61  stddev:0.14 (3.1%)  stddev-sans-first:0.15
> Values: 4.6  4.6  4.7  4.6  5.1  4.6  4.7  4.6  4.6  4.7  4.5  4.7  4.6  4.6  4.6
> 
> [#7] icon-open-DPI1.all.TART  Cycles:15  Average:6.00  Median:5.94  stddev:0.16 (2.6%)  stddev-sans-first:0.16
> Values: 5.8  5.9  6.1  5.8  6.4  5.9  5.9  6.0  5.9  6.0  6.0  6.3  6.1  5.9  5.9
> 
> [#8] icon-open-DPI1.error.TART  Cycles:15  Average:32.84  Median:32.31  stddev:1.62 (5.0%)  stddev-sans-first:1.63
> Values: 34.3  31.9  32.0  33.0  32.0  32.2  29.9  33.4  31.9  30.8  35.1  34.7  35.8  32.3  33.3
> 
> [#9] icon-close-DPI1.half.TART  Cycles:15  Average:5.65  Median:5.65  stddev:0.16 (2.8%)  stddev-sans-first:0.16
> Values: 5.6  5.4  5.4  5.4  5.6  5.6  5.6  5.7  5.7  5.8  5.9  5.7  5.7  5.8  5.8
> 
> [#10] icon-close-DPI1.all.TART  Cycles:15  Average:6.40  Median:6.34  stddev:0.29 (4.6%)  stddev-sans-first:0.30
> Values: 6.3  6.1  6.1  6.0  6.1  6.3  6.2  6.3  6.5  6.7  7.0  6.4  6.6  6.5  6.7
> 
> [#11] icon-close-DPI1.error.TART  Cycles:15  Average:6.40  Median:5.97  stddev:1.82 (30.4%)  stddev-sans-first:1.88
> Values: 6.9  4.1  3.6  6.6  5.8  6.0  5.1  8.4  8.7  8.3  3.9  5.6  5.9  8.0  9.1
> 
> [#12] icon-open-DPI2.half.TART  Cycles:15  Average:5.57  Median:5.67  stddev:0.21 (3.7%)  stddev-sans-first:0.22
> Values: 5.5  5.2  5.1  5.3  5.6  5.7  5.7  5.6  5.7  5.7  5.8  5.6  5.7  5.7  5.7
> 
> [#13] icon-open-DPI2.all.TART  Cycles:15  Average:7.13  Median:7.29  stddev:0.28 (3.8%)  stddev-sans-first:0.28
> Values: 6.8  6.9  6.4  6.8  7.0  7.3  7.3  7.2  7.4  7.3  7.3  7.3  7.3  7.1  7.3
> 
> [#14] icon-open-DPI2.error.TART  Cycles:15  Average:36.56  Median:36.29  stddev:1.77 (4.9%)  stddev-sans-first:1.74
> Values: 38.5  39.8  34.4  37.9  37.4  35.2  36.3  39.2  37.4  36.5  34.6  36.0  34.6  36.1  34.5
> 
> [#15] icon-close-DPI2.half.TART  Cycles:15  Average:5.82  Median:5.94  stddev:0.29 (4.9%)  stddev-sans-first:0.28
> Values: 5.4  5.4  5.2  5.5  5.8  5.9  5.9  6.0  6.0  6.1  6.1  6.1  6.0  5.9  6.1
> 
> [#16] icon-close-DPI2.all.TART  Cycles:15  Average:6.58  Median:6.71  stddev:0.44 (6.5%)  stddev-sans-first:0.43
> Values: 6.1  5.9  5.9  6.1  6.4  6.5  6.5  6.9  6.9  6.9  7.3  6.7  7.1  6.7  6.8
> 
> [#17] icon-close-DPI2.error.TART  Cycles:15  Average:7.57  Median:7.31  stddev:2.55 (34.9%)  stddev-sans-first:2.55
> Values: 5.1  14.6  7.3  4.1  5.4  8.2  7.1  8.6  8.2  6.8  7.5  8.4  9.2  8.9  4.3
> 
> [#18] iconFade-close-DPI2.half.TART  Cycles:15  Average:5.85  Median:5.95  stddev:0.33 (5.5%)  stddev-sans-first:0.30
> Values: 5.3  5.5  5.3  5.4  5.8  5.9  5.9  5.9  6.0  6.1  6.1  6.3  6.2  6.0  6.1
> 
> [#19] iconFade-close-DPI2.all.TART  Cycles:15  Average:5.87  Median:5.98  stddev:0.30 (4.9%)  stddev-sans-first:0.27
> Values: 5.4  5.5  5.4  5.4  5.8  6.0  5.9  5.9  6.1  6.1  6.1  6.2  6.2  6.0  6.1
> 
> [#20] iconFade-close-DPI2.error.TART  Cycles:15  Average:2.52  Median:1.64  stddev:1.73 (105.8%)  stddev-sans-first:1.75
> Values: 1.1  1.1  4.1  1.6  3.4  3.2  5.4  0.4  4.7  0.7  1.0  1.0  3.8  4.7  1.5
> 
> [#21] iconFade-open-DPI2.half.TART  Cycles:15  Average:5.60  Median:5.66  stddev:0.29 (5.2%)  stddev-sans-first:0.28
> Values: 5.2  5.2  5.3  5.2  5.6  5.7  5.7  6.3  5.7  5.7  5.6  5.8  5.7  5.8  5.7
> 
> [#22] iconFade-open-DPI2.all.TART  Cycles:15  Average:5.91  Median:6.00  stddev:0.34 (5.7%)  stddev-sans-first:0.32
> Values: 5.4  5.5  5.5  5.4  6.1  5.9  6.0  6.5  5.9  6.0  6.0  6.2  6.1  6.3  6.0
> 
> [#23] iconFade-open-DPI2.error.TART  Cycles:15  Average:2.91  Median:2.95  stddev:1.53 (52.1%)  stddev-sans-first:1.52
> Values: 1.2  0.3  1.8  1.0  1.8  4.9  4.0  4.4  2.2  2.9  3.7  4.3  1.9  4.8  4.3
> 
> [#24] newtab-open-preload-no.half.TART  Cycles:15  Average:8.12  Median:7.68  stddev:1.18 (15.4%)  stddev-sans-first:0.52
> Values: 12.0  7.7  7.3  7.9  7.8  7.7  7.6  8.4  9.2  8.3  8.1  7.4  7.4  7.3  7.7
> 
> [#25] newtab-open-preload-no.all.TART  Cycles:15  Average:9.68  Median:9.15  stddev:1.22 (13.3%)  stddev-sans-first:0.40
> Values: 13.8  9.0  9.2  9.3  9.5  9.5  9.4  9.7  10.3  9.7  9.6  9.0  9.1  8.8  9.0
> 
> [#26] newtab-open-preload-no.error.TART  Cycles:15  Average:35.07  Median:34.81  stddev:2.54 (7.3%)  stddev-sans-first:2.63
> Values: 34.8  34.3  38.1  32.8  39.5  38.9  36.1  33.9  31.1  35.4  31.3  33.5  36.1  36.7  33.7
> 
> [#27] newtab-open-preload-yes.half.TART  Cycles:15  Average:4.72  Median:4.70  stddev:0.13 (2.9%)  stddev-sans-first:0.14
> Values: 4.6  4.7  5.0  4.5  4.6  4.9  4.7  4.9  4.7  4.8  4.6  4.7  4.7  4.7  4.6
> 
> [#28] newtab-open-preload-yes.all.TART  Cycles:15  Average:6.49  Median:6.42  stddev:0.19 (2.9%)  stddev-sans-first:0.17
> Values: 6.8  6.3  6.8  6.4  6.2  6.3  6.4  6.4  6.7  6.6  6.4  6.5  6.4  6.7  6.3
> 
> [#29] newtab-open-preload-yes.error.TART  Cycles:15  Average:65.32  Median:64.43  stddev:4.05 (6.3%)  stddev-sans-first:2.84
> Values: 76.1  68.6  63.9  67.9  64.2  62.9  64.5  57.3  64.2  64.0  66.7  66.0  67.1  64.4  61.8
> -------- Summary: end --------

I'd say that the numbers do show that some of the tests regress slightly around the 4th replicate. I guess somehow it's harder for the browser to animate the tabs while the notification is displayed.

With that in mind, once we noticed that the regression seems to start with the notification, I'd say go ahead, land the talos patch to disable the notification, and let's see that it "fixes" the regression.

gfritzsche, next time I expect you to do this work ;)
Ping Joel or myself if you need any help.
Attachment #8482298 - Flags: feedback- → feedback+
Ok, thanks for looking into this Avi - with not having too much time to spare and this patch definitely missing on Talos i rather wanted this to land first.
all that is left is to modify talos.json in the main tree:
http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json

change the revision it references:
http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json#8

to be what you just landed '849898ecb9be'.  As it stands, there are no other changes in the talos repo, so there is no danger in picking up other changes.  this could be safely uplifted to aurora and beta.
Attachment #8483389 - Flags: review?(jmaher)
Comment on attachment 8483389 [details] [diff] [review]
Change talos revision

Review of attachment 8483389 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!
Attachment #8483389 - Flags: review?(jmaher) → review+
Aurora is closed right now, comment 15 still needs uplift.
Flags: needinfo?(georg.fritzsche)
(In reply to Georg Fritzsche [:gfritzsche] from comment #15)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/7a721388e46b

Hm, so this strangely seems to have triggered "WEBGL Terrain" regressions:
http://graphs.mozilla.org/graph.html#tests=[[325,131,35]]&sel=1409656834000,1409829634000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,33]]&sel=1409656425000,1409829225000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,31]]&sel=1409658214000,1409831014000&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[325,131,37]]&sel=1409658219000,1409831019000&displayrange=7&datatype=running

Those seem to look real, right?
I don't really see other possible causes except maybe the healthreporter being instantiated at a different time, so i'll have to check more into this tomorrow - jmaher pointed me to how to actually run tg1 locally.
Flags: needinfo?(georg.fritzsche)
https://hg.mozilla.org/mozilla-central/rev/7a721388e46b
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 35
Depends on: 1062826
Blocks: 1076990
Product: Firefox Health Report → Firefox Health Report Graveyard
You need to log in before you can comment on or make changes to this bug.