1420070 - 15.58 - 66.74% tp6_youtube / tp6_youtube_heavy (linux64, windows10-64, windows7-32) regression on push a40ffd441b9a724a6a11a41f282f8b69050b98c0 (Wed Nov 22 2017)

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=a40ffd441b9a724a6a11a41f282f8b69050b98c0

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 67%  tp6_youtube summary linux64 pgo e10s     231.75 -> 386.42
 64%  tp6_youtube summary linux64 pgo 1_thread e10s234.54 -> 385.12
 61%  tp6_youtube_heavy summary linux64 pgo e10s238.33 -> 384.00
 50%  tp6_youtube summary windows7-32 pgo 1_thread e10s237.21 -> 356.42
 49%  tp6_youtube_heavy summary windows10-64 pgo e10s231.17 -> 344.33
 45%  tp6_youtube summary windows10-64 pgo e10s232.79 -> 338.54
 36%  tp6_youtube_heavy summary linux64 opt e10s261.54 -> 356.08
 34%  tp6_youtube summary linux64 opt 1_thread e10s262.62 -> 352.12
 34%  tp6_youtube summary windows10-64 pgo 1_thread e10s244.75 -> 327.88
 27%  tp6_youtube summary linux64 opt e10s     268.04 -> 341.42
 25%  tp6_youtube summary windows7-32 opt e10s 280.79 -> 351.54
 17%  tp6_youtube summary windows7-32 opt 1_thread e10s294.77 -> 345.04
 16%  tp6_youtube summary windows10-64 opt e10s276.79 -> 319.92

Improvements:

 82%  tsvgr_opacity summary osx-10-10 opt e10s     330.66 -> 58.64
 79%  tsvgr_opacity summary windows7-32 pgo e10s   234.64 -> 48.99
 79%  tsvgr_opacity summary linux64 opt e10s       374.31 -> 78.84
 78%  tsvgr_opacity summary linux64 pgo e10s       336.88 -> 75.03
 77%  tsvgr_opacity summary windows7-32 opt e10s   277.87 -> 63.63
 76%  tsvgr_opacity summary windows10-64 opt e10s  232.74 -> 56.46
 74%  tsvgr_opacity summary windows10-64 pgo e10s  216.50 -> 55.79
 66%  tp6_amazon summary osx-10-10 opt 1_thread e10s957.38 -> 327.79
 65%  tp6_amazon_heavy summary osx-10-10 opt e10s  1,009.67 -> 350.92
 62%  tp6_amazon summary osx-10-10 opt e10s        914.42 -> 346.83
 54%  tp6_amazon summary linux64 opt 1_thread e10s 758.25 -> 347.17
 54%  tp6_amazon summary linux64 opt e10s          750.75 -> 345.54
 50%  tp6_amazon summary windows7-32 opt e10s      705.00 -> 351.79
 50%  tp6_amazon_heavy summary linux64 opt e10s    704.33 -> 352.12
 50%  tp6_amazon summary windows10-64 opt 1_thread e10s626.67 -> 313.67
 50%  tp6_amazon summary windows7-32 opt 1_thread e10s680.17 -> 341.62
 50%  tp6_amazon_heavy summary windows10-64 opt e10s636.83 -> 320.62
 48%  tp6_amazon summary linux64 pgo 1_thread e10s 715.06 -> 373.62
 47%  tp6_amazon summary windows10-64 opt e10s     608.67 -> 319.62
 46%  tp6_amazon summary windows7-32 pgo e10s      578.88 -> 310.58
 46%  tp6_amazon summary linux64 pgo e10s          705.00 -> 382.04
 42%  tp6_amazon_heavy summary linux64 pgo e10s    680.54 -> 391.71
 40%  tp6_amazon summary windows7-32 pgo 1_thread e10s582.92 -> 349.25
 39%  tp6_amazon summary windows10-64 pgo 1_thread e10s536.54 -> 326.58
 35%  tp6_amazon_heavy summary windows10-64 pgo e10s535.38 -> 345.42
 35%  tp6_amazon summary windows10-64 pgo e10s     522.00 -> 337.25
 25%  tp6_google summary osx-10-10 opt e10s        442.42 -> 330.12
 24%  tp6_google summary osx-10-10 opt 1_thread e10s432.96 -> 329.88
 24%  tp6_google summary linux64 opt e10s          462.46 -> 352.79
 22%  tp6_google summary windows10-64 opt 1_thread e10s408.17 -> 318.42
 22%  tp6_google_heavy summary linux64 opt e10s    459.08 -> 359.88
 21%  tp6_google summary windows10-64 opt e10s     406.33 -> 319.79
 21%  tp6_google summary linux64 opt 1_thread e10s 440.17 -> 346.50
 21%  tp6_google_heavy summary windows10-64 opt e10s405.12 -> 319.71
 18%  tp6_google summary windows7-32 opt 1_thread e10s421.54 -> 346.71
 17%  tp6_google summary windows7-32 opt e10s      421.33 -> 349.79
 16%  tp6_google_heavy summary osx-10-10 opt e10s  423.08 -> 356.08
  8%  tp6_google_heavy summary linux64 pgo e10s    414.81 -> 380.79
  8%  tp6_google summary linux64 pgo e10s          419.92 -> 385.75
  6%  tp5o_webext responsiveness windows10-64 opt e10s3.64 -> 3.41
  6%  tp5o_webext responsiveness windows10-64 pgo e10s3.42 -> 3.22


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=10691

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Robert, unless there's something strange about any of the results above, please resolve this as WONTFIX.

Comment 2

[Robert Wood [:rwood]]

Wow.. no this is not expected, maybe slight improvements if anything but not this extreme... I'm having a look, thanks!

Comment 3

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5cf0db6a6126b29bb9ba520f83befea36d356ba

Comment 4

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7989796d8149fac74fb020e8afa113c2d4e547a

Comment 5

[Robert Wood [:rwood]]

Oh man, I see the issue, it's a manifest problem... 

14:02:07     INFO -  Initialising browser for tp6_youtube test...
14:02:07     INFO -  Cloning profile located at /builds/slave/test/build/tests/talos/talos/base_profile
14:02:07     INFO -  Installing Add-ons
14:02:07     INFO -  Application command: /builds/slave/test/build/application/firefox/firefox  http://localhost:39023/getInfo.html -profile /tmp/tmpEiUGVL/profile
14:02:08     INFO -  TEST-INFO | started process 7640 (/builds/slave/test/build/application/firefox/firefox  http://localhost:39023/getInfo.html)
14:02:12     INFO -  127.0.0.1:34883: clientconnect
14:02:12     INFO -  For request https://www.facebook.com/ best match https:/

Note that this is tp6_youtube, however it is loading www.facebook.com (from the mitmproxy archive). Trying to figure out why - will work on a fix ASAP.

Comment 6

[Robert Wood [:rwood]]

Attachment: Bug 1420070 - Fix talos pageloader manifest bug;

Review commit: https://reviewboard.mozilla.org/r/202896/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/202896/

Comment 7

[Robert Wood [:rwood]]

That was a bad bug.

The talos.* browser preferences were being set in the wrong run_tests test loop - they were being set where the info was being retrieved and therefore overwriting eachother each time.

Then when it got to the second run_tests loop - where it actually runs each test, only the talos.* browser preferences for the last test in the list were being used for that test; therefore only the manifest from the last test in the list was being run.

Fix is to set the talos.* browser preferences in the correct run_tests loop - right before each test / browser starts running, guaranteeing the correct prefs are set for each test in the suite.

Also the 'talos.tpmanifest' pref wasn't being set in the set preferences function; it was only being set via ffsetup. Fix was to add 'tpmanifest' to list of items in set_tp_preferences.

Comment 8

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f13709b1bc7702201655ad96885e3b8c623b8207

Comment 9

[Robert Wood [:rwood]]

Comment on attachment 8931772 [details]
Bug 1420070 - Fix talos pageloader manifest bug;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/202896/diff/1-2/

Comment 10

[Andrew Halberstadt [:ahal]]

Comment on attachment 8931772 [details]
Bug 1420070 - Fix talos pageloader manifest bug;

https://reviewboard.mozilla.org/r/202896/#review208230

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2f63ef51f61
Fix talos pageloader manifest bug; r=ahal

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/f2f63ef51f61

Comment 13

[Ionuț Goldan [:igoldan]]

Almost all results have returned to normal, with the exception on the tp5o_webext ones.

== Change summary for alert #10733 (as of Fri, 24 Nov 2017 20:31:16 GMT) ==

Regressions:

472%  tsvgr_opacity summary osx-10-10 opt e10s     58.44 -> 334.12
374%  tsvgr_opacity summary linux64 opt e10s       79.11 -> 374.90
360%  tsvgr_opacity summary windows7-32 pgo e10s   48.99 -> 225.13
346%  tsvgr_opacity summary linux64 pgo e10s       75.32 -> 335.62
331%  tsvgr_opacity summary windows7-32 opt e10s   64.07 -> 276.04
304%  tsvgr_opacity summary windows10-64 opt e10s  57.82 -> 233.31
291%  tsvgr_opacity summary windows10-64 pgo e10s  55.87 -> 218.23
192%  tp6_amazon summary osx-10-10 opt 1_thread e10s323.04 -> 942.88
186%  tp6_amazon summary osx-10-10 opt e10s        337.17 -> 965.71
185%  tp6_amazon_heavy summary osx-10-10 opt e10s  357.17 -> 1,017.92
119%  tp6_amazon summary linux64 opt e10s          347.88 -> 762.12
116%  tp6_amazon summary linux64 opt 1_thread e10s 348.58 -> 754.04
104%  tp6_amazon summary windows7-32 opt 1_thread e10s342.96 -> 698.88
 96%  tp6_amazon summary windows7-32 opt e10s      347.38 -> 680.71
 95%  tp6_amazon_heavy summary linux64 opt e10s    360.92 -> 704.29
 94%  tp6_amazon summary linux64 pgo e10s          380.71 -> 738.25
 94%  tp6_amazon summary windows7-32 pgo e10s      291.92 -> 565.42
 92%  tp6_amazon_heavy summary windows10-64 opt e10s322.17 -> 619.62
 91%  tp6_amazon summary linux64 pgo 1_thread e10s 376.38 -> 720.58
 89%  tp6_amazon summary windows10-64 opt 1_thread e10s320.54 -> 604.58
 84%  tp6_amazon summary windows10-64 opt e10s     329.54 -> 604.71
 74%  tp6_amazon_heavy summary linux64 pgo e10s    385.21 -> 668.79
 65%  tp6_amazon summary windows7-32 pgo 1_thread e10s353.96 -> 583.25
 62%  tp6_amazon_heavy summary windows10-64 pgo e10s326.62 -> 530.46
 60%  tp6_amazon summary windows10-64 pgo 1_thread e10s329.67 -> 528.62
 51%  tp6_amazon summary windows10-64 pgo e10s     343.46 -> 517.71
 35%  tp6_google summary osx-10-10 opt 1_thread e10s327.94 -> 441.92
 34%  tp6_google summary linux64 opt e10s          346.46 -> 464.92
 32%  tp6_google summary osx-10-10 opt e10s        337.92 -> 444.75
 28%  tp6_google summary linux64 opt 1_thread e10s 345.25 -> 441.17
 27%  tp6_google_heavy summary linux64 opt e10s    363.50 -> 460.46
 26%  tp6_google summary windows10-64 opt e10s     320.88 -> 405.42
 26%  tp6_google_heavy summary windows10-64 opt e10s321.75 -> 405.75
 25%  tp6_google_heavy summary osx-10-10 opt e10s  347.25 -> 435.17
 25%  tp6_google summary windows10-64 opt 1_thread e10s324.96 -> 406.79
 21%  tp6_google summary windows7-32 opt 1_thread e10s348.67 -> 421.46
 19%  tp6_google summary windows7-32 opt e10s      351.92 -> 419.21
 11%  tp6_google_heavy summary linux64 pgo e10s    378.31 -> 418.12
 10%  tp6_google summary linux64 pgo e10s          382.50 -> 420.71
  6%  tp6_google summary windows10-64 pgo 1_thread e10s326.69 -> 347.83
  5%  tp6_google summary linux64 pgo 1_thread e10s 385.90 -> 403.29

Improvements:

 39%  tp6_youtube_heavy summary linux64 pgo e10s     387.92 -> 236.71
 38%  tp6_youtube summary linux64 pgo 1_thread e10s  385.88 -> 238.79
 38%  tp6_youtube summary linux64 pgo e10s           382.04 -> 237.92
 34%  tp6_youtube summary windows7-32 pgo 1_thread e10s333.67 -> 220.92
 33%  tp6_youtube summary windows10-64 pgo e10s      343.42 -> 230.71
 32%  tp6_youtube_heavy summary windows10-64 pgo e10s339.35 -> 230.33
 26%  tp6_youtube summary windows7-32 pgo e10s       290.85 -> 214.21
 24%  tp6_youtube_heavy summary linux64 opt e10s     356.75 -> 269.42
 24%  tp6_youtube summary linux64 opt e10s           349.25 -> 263.83
 23%  tp6_youtube summary linux64 opt 1_thread e10s  347.29 -> 266.08
 23%  tp6_youtube summary windows10-64 pgo 1_thread e10s329.96 -> 253.21
 21%  tp6_youtube summary windows7-32 opt e10s       351.62 -> 276.38
 17%  tp6_youtube summary windows7-32 opt 1_thread e10s349.46 -> 288.71
 10%  tp6_youtube summary windows10-64 opt e10s      324.52 -> 290.88

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=10733