1434233 - 40.27 - 797.34% glvideo Mean tick time across 100 ticks: / stylebench / tp5o XRes / tp5o_webext XRes (linux64) regression on push 5e78fa0f7d18 (Mon Jan 29 2018)

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=873831981ed89de7b81ccdacba966d206cc3b701&tochange=5e78fa0f7d18b0abd9f54d01b057d26c49499c9d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

797%  tp5o_webext XRes linux64 pgo e10s     723,873.57 -> 6,495,583.29
778%  tp5o XRes linux64 pgo e10s            739,866.68 -> 6,494,279.49
669%  tp5o XRes linux64 opt e10s            844,360.14 -> 6,495,291.55
428%  tp5o_webext XRes linux64 opt e10s     1,229,736.83 -> 6,496,375.30
122%  glvideo Mean tick time across 100 ticks:  linux64 opt e10s6.57 -> 14.60
122%  glvideo Mean tick time across 100 ticks:  linux64 pgo e10s6.58 -> 14.59
 44%  stylebench linux64 opt e10s           16.69 -> 9.43
 40%  stylebench linux64 pgo e10s           14.86 -> 8.87

Improvements:

101%  dromaeo_css linux64 pgo e10s     8,947.48 -> 18,023.25
101%  dromaeo_css linux64 opt e10s     8,410.72 -> 16,877.45
 98%  speedometer linux64 opt e10s     34.56 -> 68.55
 90%  tp5o_scroll linux64 pgo e10s     5.91 -> 0.60
 90%  tp5o_scroll linux64 opt e10s     6.01 -> 0.62
 89%  speedometer linux64 pgo e10s     40.65 -> 76.94
 87%  tscrollx linux64 pgo e10s        6.28 -> 0.81
 87%  tscrollx linux64 opt e10s        6.53 -> 0.86
 84%  rasterflood_gradient linux64 opt e10s169.00 -> 310.75
 76%  dromaeo_dom linux64 opt e10s     2,099.32 -> 3,692.66
 73%  tp5o responsiveness linux64 pgo e10s2.57 -> 0.70
 72%  dromaeo_dom linux64 pgo e10s     2,269.99 -> 3,907.04
 72%  tp5o responsiveness linux64 opt e10s2.92 -> 0.83
 69%  rasterflood_gradient linux64 pgo e10s169.58 -> 286.67
 69%  tp5o_webext responsiveness linux64 opt e10s6.17 -> 1.92
 69%  tp5o_webext responsiveness linux64 pgo e10s5.44 -> 1.71
 59%  tp6_amazon linux64 opt e10s      778.33 -> 315.42
 59%  tp6_amazon linux64 pgo e10s      705.12 -> 288.42
 58%  tp6_amazon linux64 opt 1_thread e10s751.21 -> 313.12
 58%  tp6_amazon linux64 pgo 1_thread e10s673.46 -> 284.38
 58%  a11yr linux64 opt e10s           569.78 -> 242.02
 56%  tresize linux64 opt e10s         28.50 -> 12.47
 56%  tresize linux64 pgo e10s         27.96 -> 12.36
 55%  tp6_amazon_heavy linux64 pgo e10s632.96 -> 286.00
 55%  a11yr linux64 pgo e10s           440.95 -> 199.88
 54%  tp6_amazon_heavy linux64 opt e10s692.04 -> 317.88
 54%  tp6_facebook linux64 pgo e10s    367.23 -> 169.50
 52%  tp6_facebook linux64 pgo 1_thread e10s357.08 -> 169.92
 52%  tart linux64 opt e10s            4.79 -> 2.30
 52%  basic_compositor_video linux64 opt e10s5.58 -> 2.70
 52%  basic_compositor_video linux64 pgo e10s5.57 -> 2.70
 51%  tart linux64 pgo e10s            4.31 -> 2.09
 51%  ts_paint_webext linux64 pgo e10s 1,082.08 -> 526.42
 51%  glterrain linux64 opt e10s       11.40 -> 5.56
 51%  tp6_google linux64 opt e10s      920.88 -> 452.08
 51%  tp6_google linux64 opt 1_thread e10s900.58 -> 443.21
 51%  ts_paint_webext linux64 opt e10s 1,136.25 -> 561.92
 50%  tp6_facebook linux64 opt 1_thread e10s360.67 -> 179.79
 50%  glterrain linux64 pgo e10s       11.03 -> 5.55
 50%  tp6_google linux64 pgo e10s      834.46 -> 421.21
 49%  tp6_google linux64 pgo 1_thread e10s809.54 -> 414.00
 48%  tp6_facebook linux64 opt e10s    348.50 -> 180.25
 48%  tp6_facebook_heavy linux64 opt e10s345.42 -> 179.54
 48%  tp6_facebook_heavy linux64 pgo e10s320.19 -> 167.71
 46%  tsvgr_opacity linux64 opt e10s   369.95 -> 200.81
 46%  perf_reftest linux64 opt e10s    3.10 -> 1.69
 45%  tp5o linux64 opt e10s            282.84 -> 155.74
 45%  tsvgr_opacity linux64 pgo e10s   335.32 -> 184.89
 44%  tp5o_webext linux64 opt e10s     410.05 -> 228.82
 43%  tp5o linux64 pgo e10s            246.27 -> 140.71
 42%  tp5o_webext linux64 pgo e10s     364.81 -> 209.87
 42%  ts_paint_heavy linux64 opt e10s  803.42 -> 462.42
 42%  kraken linux64 opt e10s          1,406.88 -> 810.80
 42%  tsvgx linux64 opt e10s           376.46 -> 217.10
 42%  kraken linux64 pgo e10s          1,366.32 -> 791.88
 42%  tsvg_static linux64 opt e10s     79.16 -> 45.89
 42%  tpaint linux64 opt e10s          246.04 -> 142.78
 41%  tsvg_static linux64 pgo e10s     75.36 -> 44.25
 41%  perf_reftest_singletons linux64 opt e10s28.15 -> 16.55
 41%  ts_paint linux64 opt e10s        764.58 -> 451.00
 40%  tsvgx linux64 pgo e10s           354.25 -> 211.65
 40%  perf_reftest_singletons linux64 pgo e10s26.19 -> 15.67
 40%  ts_paint_heavy linux64 pgo e10s  718.08 -> 434.42
 39%  perf_reftest linux64 pgo e10s    2.83 -> 1.72
 39%  tpaint linux64 pgo e10s          220.30 -> 134.00
 39%  sessionrestore_no_auto_restore linux64 opt e10s547.08 -> 333.50
 39%  sessionrestore linux64 opt e10s  462.50 -> 282.00
 39%  ts_paint linux64 pgo e10s        690.58 -> 424.17
 38%  sessionrestore linux64 pgo e10s  423.75 -> 264.17
 37%  sessionrestore_no_auto_restore linux64 pgo e10s495.17 -> 310.17
 37%  displaylist_mutate linux64 pgo e10s4,399.42 -> 2,775.07
 36%  damp linux64 opt e10s            80.81 -> 51.96
 35%  sessionrestore_many_windows linux64 opt e10s1,417.00 -> 916.25
 35%  tps linux64 opt e10s             28.45 -> 18.44
 35%  tps linux64 pgo e10s             26.96 -> 17.55
 35%  displaylist_mutate linux64 opt e10s4,923.46 -> 3,220.44
 34%  damp linux64 pgo e10s            73.41 -> 48.17
 33%  rasterflood_svg linux64 opt e10s 26,297.78 -> 17,745.38
 33%  sessionrestore_many_windows linux64 pgo e10s1,251.08 -> 844.33
 32%  rasterflood_svg linux64 pgo e10s 27,448.45 -> 18,684.06
 31%  tp6_youtube linux64 opt 1_thread e10s273.96 -> 188.46
 30%  tp6_youtube_heavy linux64 opt e10s268.42 -> 186.71
 30%  tabpaint linux64 pgo e10s        71.52 -> 49.96
 29%  tp6_youtube linux64 opt e10s     267.08 -> 189.71
 27%  tabpaint linux64 opt e10s        72.45 -> 52.97
 26%  tp6_youtube_heavy linux64 pgo e10s237.28 -> 174.88
 24%  cpstartup content-process-startup linux64 opt e10s265.92 -> 201.42
 24%  tp6_youtube linux64 pgo e10s     237.83 -> 181.17
 21%  cpstartup content-process-startup linux64 pgo e10s249.25 -> 195.92
  3%  tp6_google_heavy linux64 opt e10s462.12 -> 448.17


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=11323

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Lots of improvements after bug 1433702 landed. My question: were the regressions expected?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we saw these exact regressions earlier on, basically the ones of concern are:
* glvideo (in need of an owner)
* stylebench  (this is new and there might be adjustments needed to the teest)

I see that :jgilbert reviewed the patch to land glvideo in bug 1355726- maybe he could comment on if it makes sense that switching hardware (specifically from nvidia -> intel) would cause glvideo to regress.

:emilio has helped with information related to stylebench- I want his opinion on concerns here- it is odd that new hardware would cause this, but maybe it depends on webGL and there is an issue with nvidia vs intel.

Comment 3

[Kelsey Gilbert [:jgilbert]]

Regressions or improvements on any graphics workloads from switching GPUs is expected, not just nv->intel.
This looks fine to me.

Comment 4

[Emilio Cobos Álvarez (:emilio)]

StyleBench does not depend on WebGL in any way, so I'd expect it to improve. Let's sort the stylebench issues in bug 1425058.

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Emilio Cobos Álvarez [:emilio] from comment #4)
> StyleBench does not depend on WebGL in any way, so I'd expect it to improve.
> Let's sort the stylebench issues in bug 1425058.

Marking this as resolved, as stylebench improved a lot [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1425058#c18