1477638 - 2.48 - 65.01% displaylist_mutate / tabpaint / tp5o*_* / tp6_* / tresize / tscrollx / tsvg_static / tsvgr_opacity / tsvgx (linux64-qr, windows10-64-qr) regression on push 80efe717b563f329a2261a8d23f4ebbc470c432f (Sat Jul 21 2018)

Status: RESOLVED WONTFIX

(Core :: Graphics: WebRender, defect, P4)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=80efe717b563f329a2261a8d23f4ebbc470c432f

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 65%  tp5o_scroll windows10-64-qr opt e10s stylo     0.44 -> 0.73
 38%  tp5o_scroll linux64-qr opt e10s stylo          0.53 -> 0.74
 22%  tsvgx linux64-qr opt e10s stylo                349.39 -> 425.19
 21%  tsvgx windows10-64-qr opt e10s stylo           389.04 -> 472.42
 18%  tsvg_static linux64-qr opt e10s stylo          56.51 -> 66.48
 12%  tp5o responsiveness linux64-qr opt e10s stylo  1.26 -> 1.41
 10%  tp5o_webext responsiveness linux64-qr opt e10s stylo2.26 -> 2.49
  9%  tscrollx linux64-qr opt e10s stylo             0.42 -> 0.46
  8%  tsvgr_opacity windows10-64-qr opt e10s stylo   103.76 -> 112.33
  7%  tresize linux64-qr opt e10s stylo              13.37 -> 14.30
  6%  tp5o_webext linux64-qr opt e10s stylo          239.76 -> 254.39
  6%  tp5o linux64-qr opt e10s stylo                 151.99 -> 160.41
  5%  tabpaint linux64-qr opt e10s stylo             52.14 -> 54.62
  4%  tp6_amazon linux64-qr opt e10s stylo           283.71 -> 296.46
  4%  displaylist_mutate linux64-qr opt e10s stylo   4,795.40 -> 4,971.36
  4%  displaylist_mutate windows10-64-qr opt e10s stylo3,941.65 -> 4,085.30
  3%  tp6_google linux64-qr opt e10s stylo           458.08 -> 469.75
  2%  tp6_google_heavy linux64-qr opt e10s stylo     455.54 -> 466.83

Improvements:

 39%  tsvg_static osx-10-10 opt e10s stylo     86.97 -> 53.00
 29%  tsvg_static windows7-32 pgo e10s stylo   73.30 -> 52.19
 27%  tsvg_static windows7-32 opt e10s stylo   72.74 -> 52.75
 24%  tscrollx windows10-64-qr opt e10s stylo  0.48 -> 0.36
 23%  tsvg_static linux64 pgo e10s stylo       70.15 -> 53.83
 23%  tsvg_static linux64 opt e10s stylo       71.13 -> 54.68


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=14486

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

These regressions were caused by this update: https://bugzilla.mozilla.org/show_bug.cgi?id=1476636#c8

Comment 2

[Jeff Muizelaar [:jrmuizel]]

It seems likely that at least some of this is caused by async blob.

Comment 3

[Kartikaya Gupta (email:[e-mail]) (away until 17-Feb-2020)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #0)
> Improvements:
> 
>  39%  tsvg_static osx-10-10 opt e10s stylo     86.97 -> 53.00
>  29%  tsvg_static windows7-32 pgo e10s stylo   73.30 -> 52.19
>  27%  tsvg_static windows7-32 opt e10s stylo   72.74 -> 52.75
>  24%  tscrollx windows10-64-qr opt e10s stylo  0.48 -> 0.36
>  23%  tsvg_static linux64 pgo e10s stylo       70.15 -> 53.83
>  23%  tsvg_static linux64 opt e10s stylo       71.13 -> 54.68

The tsvg_static stuff here is unrelated, I think. From the perfherder graphs it looks like it's from bug 1477260.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

I'm surprised that displaylist_mutate would have become worse. Are we sure that was caused by this change?

Comment 5

[Kartikaya Gupta (email:[e-mail]) (away until 17-Feb-2020)]

The inbound range is https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=ae7ccb8c938d180f5777d4be8885cd425b711966&tochange=558782eeafe1ceb6bf672afe1bd0ef80e62b7cbb and the autoland range is https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9daa53881b7ae80bf6b093dac5d7744cf7fd18b1&tochange=0efbf75ea1b67c99f285517d5f5235b7f98162e4 so it's pretty likely it's from the same change. I've triggered the missing talos jobs on the inbound changeset so we'll know for sure once those are done.

Comment 6

[Kartikaya Gupta (email:[e-mail]) (away until 17-Feb-2020)]

Confirmed: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&group_state=expanded&tochange=80efe717b563f329a2261a8d23f4ebbc470c432f&filter-searchStr=Linux%20x64%20QuantumRender%20opt%20Talos%20performance%20tests%20with%20e10s%20test-linux64-qr%2Fopt-talos-g4-e10s%20T-e10s(g4)&fromchange=ae7ccb8c938d180f5777d4be8885cd425b711966

I also checked the try pushes from the bug 1476636 comment 1 which doesn't have the regression and bug 1476636 comment 4 which does. And that produces a regression range of this:

* 3baeb234 Auto merge of #2785 - nical:async-blob, r=kvark
* 6b29ecc8 Rasterize blob images on the scene builder thread.

So it's definitely from the async blob changes.

Comment 7

[Jeff Muizelaar [:jrmuizel]]

This caused by bug 1477783

Comment 8

[Ionuț Goldan [:igoldan]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)
> (In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #0)
> > Improvements:
> > 
> >  39%  tsvg_static osx-10-10 opt e10s stylo     86.97 -> 53.00
> >  29%  tsvg_static windows7-32 pgo e10s stylo   73.30 -> 52.19
> >  27%  tsvg_static windows7-32 opt e10s stylo   72.74 -> 52.75
> >  24%  tscrollx windows10-64-qr opt e10s stylo  0.48 -> 0.36
> >  23%  tsvg_static linux64 pgo e10s stylo       70.15 -> 53.83
> >  23%  tsvg_static linux64 opt e10s stylo       71.13 -> 54.68
> 
> The tsvg_static stuff here is unrelated, I think. From the perfherder graphs
> it looks like it's from bug 1477260.

Yes, you got that right. Thank you for making the correction.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

Bug 1478135 is another known cause of this regression. I suspect that it might be a large contributor to the svg talos time regressions.

Comment 10

[Ionuț Goldan [:igoldan]]

Most of the regressions from comment 0 have been reverted, since https://bugzilla.mozilla.org/show_bug.cgi?id=1477505#c4

== Change summary for alert #14544 (as of Tue, 24 Jul 2018 00:49:10 GMT) ==

Regressions:

 29%  tscrollx windows10-64-qr opt e10s stylo     0.37 -> 0.47

Improvements:

 40%  tp5o_scroll windows10-64-qr opt e10s stylo     0.74 -> 0.44
 28%  tp5o_scroll linux64-qr opt e10s stylo          0.73 -> 0.53
 13%  tsvg_static linux64-qr opt e10s stylo          66.58 -> 57.79
 10%  tp5o_webext responsiveness linux64-qr opt e10s stylo2.43 -> 2.17
  9%  tresize windows10-64-qr opt e10s stylo         9.13 -> 8.32
  8%  tscrollx linux64-qr opt e10s stylo             0.46 -> 0.42
  7%  tp5o_webext linux64-qr opt e10s stylo          251.24 -> 233.56
  7%  tresize linux64-qr opt e10s stylo              14.34 -> 13.38
  6%  tp5o linux64-qr opt e10s stylo                 159.64 -> 149.68
  4%  tp6_facebook linux64-qr opt 1_thread e10s stylo190.02 -> 182.00
  2%  tsvgx linux64-qr opt e10s stylo                424.75 -> 414.88

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14544

Comment 11

[Ionuț Goldan [:igoldan]]

The regressions which remained are. I'll recheck the displaylist_mutate regression, to also confirm (or not) its link relation with this bug.

== Change summary for alert #14486 (as of Mon, 23 Jul 2018 05:34:05 GMT) ==

Regressions:

 22%  tsvgx linux64-qr opt e10s stylo                349.39 -> 425.19
 21%  tsvgx windows10-64-qr opt e10s stylo           389.04 -> 472.42
  8%  tsvgr_opacity windows10-64-qr opt e10s stylo   103.76 -> 112.33
  4%  displaylist_mutate windows10-64-qr opt e10s stylo3,941.65 -> 4,085.30

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14486

Comment 12

[Ionuț Goldan [:igoldan]]

Some of them improved, but only partially.

Comment 13

[Ionuț Goldan [:igoldan]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #11)
> The regressions which remained are. I'll recheck the displaylist_mutate
> regression, to also confirm (or not) its link relation with this bug.
> 
> == Change summary for alert #14486 (as of Mon, 23 Jul 2018 05:34:05 GMT) ==
> 
> Regressions:
> 
>  22%  tsvgx linux64-qr opt e10s stylo                349.39 -> 425.19
>  21%  tsvgx windows10-64-qr opt e10s stylo           389.04 -> 472.42
>   8%  tsvgr_opacity windows10-64-qr opt e10s stylo   103.76 -> 112.33
>   4%  displaylist_mutate windows10-64-qr opt e10s stylo3,941.65 -> 4,085.30
> 
> For up to date results, see:
> https://treeherder.mozilla.org/perf.html#/alerts?id=14486

The tsvgx regression were entirely fixed. Checkout bug https://bugzilla.mozilla.org/show_bug.cgi?id=1482077#c0 for more info.

Comment 14

[Ionuț Goldan [:igoldan]]

Basically, we're left with these 2 regressions:

Regressions:

  8%  tsvgr_opacity windows10-64-qr opt e10s stylo   103.76 -> 112.33
  4%  displaylist_mutate windows10-64-qr opt e10s stylo3,941.65 -> 4,085.30