1496369 - 2.42 - 40.48% a11yr / displaylist_mutate / tp5o / tp5o_scroll / tp5o_webext / tps / tscrollx (linux64, linux64-qr, windows10-64, windows10-64-qr, windows7-32) regression on push fd895bb95b99a0f0b1d8995279141773aa741703 (Wed Oct 3 2018)

Status: VERIFIED FIXED

(Core :: Panning and Zooming, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=fd895bb95b99a0f0b1d8995279141773aa741703

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 40%  tscrollx linux64 opt e10s stylo                    0.89 -> 1.25
 36%  tscrollx linux64-qr opt e10s stylo                 2.33 -> 3.17
 35%  tscrollx linux64 pgo e10s stylo                    0.85 -> 1.15
 20%  tp5o_scroll linux64-qr opt e10s stylo              3.96 -> 4.75
 18%  tps windows7-32 opt e10s stylo                     12.23 -> 14.45
 17%  tscrollx windows10-64 opt e10s stylo               0.86 -> 1.01
 15%  tp5o_scroll windows10-64-qr opt e10s stylo         2.82 -> 3.26
 15%  tp5o_scroll linux64 pgo e10s stylo                 1.07 -> 1.23
 14%  tp5o_scroll linux64 opt e10s stylo                 1.15 -> 1.32
  9%  tp5o linux64 pgo e10s stylo                        125.05 -> 135.85
  9%  tp5o linux64 opt e10s stylo                        138.76 -> 150.58
  8%  tp5o_scroll windows10-64 opt e10s stylo            1.36 -> 1.47
  7%  displaylist_mutate windows10-64 opt e10s stylo     2,165.74 -> 2,308.45
  6%  displaylist_mutate windows10-64 pgo e10s stylo     2,054.24 -> 2,181.30
  5%  a11yr windows7-32 opt e10s stylo                   183.52 -> 193.47
  5%  tp5o_webext linux64 pgo e10s stylo                 185.27 -> 195.10
  5%  tp5o_webext linux64 opt e10s stylo                 207.77 -> 217.97
  5%  displaylist_mutate windows7-32 opt e10s stylo      2,246.73 -> 2,352.68
  4%  tp5o_webext windows10-64 opt e10s stylo            192.81 -> 200.39
  4%  tp5o windows7-32 opt e10s stylo                    127.69 -> 132.42
  4%  tp5o windows10-64 opt e10s stylo                   130.29 -> 134.93
  3%  displaylist_mutate linux64 opt e10s stylo          2,025.71 -> 2,091.44
  3%  tp5o_webext windows7-32 opt e10s stylo             189.17 -> 194.83
  3%  tp5o windows10-64-qr opt e10s stylo                126.69 -> 130.07
  2%  tp5o linux64-qr opt e10s stylo                     143.86 -> 147.35


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=16404

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Here are the Gecko profiles for tp5o on Linux 64 opt:

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FebsrqnVVQKS2WrvCao-I7Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_tp5o.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FH0GYztXyRIqPuseLEM5j_Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_tp5o.zip

Comment 2

[Ionuț Goldan [:igoldan]]

And for tscrollx on the same platform:

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FUnVY9XVxRbKjInMJ1EPiqA%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_tscrollx.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FRcXchFihSZOc5S8oRvGv6Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_tscrollx.zip

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

this was backed out, so we will see improvements soon, then more regressions assuming it lands again with no adjustments to perf

Comment 4

[Botond Ballo [:botond]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> Here are the Gecko profiles for tp5o on Linux 64 opt:
> 
> before:
> https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.
> net%2Fv1%2Ftask%2FebsrqnVVQKS2WrvCao-
> I7Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_tp5o.zip
> 
> after:
> https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.
> net%2Fv1%2Ftask%2FH0GYztXyRIqPuseLEM5j_Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Fte
> st_info%2Fprofile_tp5o.zip

Whenever I try to look at one of these profiles, I get an error like this:

Are you sure this is a profile? Failed to process the file in the zip file at the following path:/profile_tp5o/stackoverflow.com_pagecycle_2/cycle_0.profile

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I just tried this and it works for me (the before link).  Maybe a hiccup in perf.html

Comment 6

[Botond Ballo [:botond]]

Some investigation of these profiles with Markus and Matt today suggested that the cause of the slowdown may be an interaction with the throttling mechanism used by tests in ASAP mode, rather than actual operations like painting or compositing taking longer.

Comment 7

[Ionuț Goldan [:igoldan]]

The backout of bug 1495055 canceled all regressions. Here is just a small subset of the perf regains:

== Change summary for alert #16514 (as of Fri, 05 Oct 2018 05:00:41 GMT) ==

Improvements:

  6%  tp5o_scroll windows10-64 opt e10s stylo           1.48 -> 1.40
  5%  displaylist_mutate windows7-32 pgo e10s stylo     2,263.83 -> 2,149.40
  5%  tp5o osx-10-10 opt e10s stylo                     229.55 -> 218.39
  3%  tp5o_scroll windows7-32 pgo e10s stylo            1.43 -> 1.38

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=16514

Comment 8

[Kartikaya Gupta (email:[e-mail])]

The relanding of that bug will be restricted in its effects to Android, so it shouldn't re-introduce these regressions. Calling this fixed by backout.