1572682 - 1.31 - 2.14% startup_about_home_paint (linux64-shippable) regression on push d780672e184a445f70d63643fce313e75d5da3c9 (Fri August 2 2019)

Status: VERIFIED FIXED

(Firefox :: New Tab Page, defect, P1)

Description

[Marian Raiciof [:marauder]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=d780672e184a445f70d63643fce313e75d5da3c9

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

2% startup_about_home_paint linux64-shippable opt e10s stylo 722.62 -> 738.08
1% startup_about_home_paint linux64-shippable opt e10s stylo 720.88 -> 730.33

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=22369

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/TestEngineering/Performance/Talos

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Kate Hudson :k88hudson]

This is likely caused by rendering extended triplets before top sites, let's take a look and see what the right thing to do here is.

Comment 2

[Ed Lee :Mardak]

Here's some quick runs of the export m-c commit:
https://perfht.ml/2ZXnFEU topsites_first_render_ts 2.205s / topsites_first_painted_ts 2.429s
https://perfht.ml/2YYTBLU topsites_first_render_ts 2.158s / topsites_first_painted_ts 2.403s
https://perfht.ml/2ZO8u0O topsites_first_render_ts 2.063s / topsites_first_painted_ts 2.284s

And reverting https://github.com/mozilla/activity-stream/commit/d40f56fd91fbad6ca9d2143a660de72b6527ae2c:
https://perfht.ml/2ZXnolm topsites_first_render_ts 2.145s / topsites_first_painted_ts 2.337s
https://perfht.ml/2Z2tSCE topsites_first_render_ts 2.080s / topsites_first_painted_ts 2.300s
https://perfht.ml/2YRekRP topsites_first_render_ts 2.122s / topsites_first_painted_ts 2.354s

At least the median number for topsites_first_painted_ts shows a 2.8% regression (this usertiming is what the talos test measures). topsites_first_render_ts median also shows a 1.7% regression for this small sample.

So quite likely it is indeed the extended triplets causing the regression.

And another sanity check of --setpref trailhead.firstrun.branches=join to not have triplets:
https://perfht.ml/2YVwUbB topsites_first_render_ts 2.189s / topsites_first_painted_ts 2.413s
https://perfht.ml/2YVwW3d topsites_first_render_ts 2.061s / topsites_first_painted_ts 2.279s
https://perfht.ml/2ZPrvjI topsites_first_render_ts 2.123s / topsites_first_painted_ts 2.343s

And the median number is within 0.5% for both markers. So showing the triplets on about:home (which this talos test uses) instead of only showing triplets on about:welcome is the source of regression.

Potentially we can run the talos test with something like branches=join to simulate the common case behavior of the profile being old enough to not show the extended triplets.

Comment 3

[Ed Lee :Mardak]

mconley, this regression is expected, but we would like to have the startup_about_home_paint measure something more accurate by having the startup test run with a different pref value that turns off "extended triplets" which only show for new users on about:home. This would "fix" the regression by having it measure the same thing it was measuring.

We would also like to have a new "triplets" test / subtest that uses the default pref value. At minimum it would also measure topsites_first_painted_ts but ideally we would (also?) measure a triplets_first_painted_ts (doesn't exist yet).

Comment 4

[Ed Lee :Mardak]

Here's a try run setting branch to join for perf:
https://hg.mozilla.org/try/rev/72f351b666684e97d8cfc26f8b43b0ef23756d0e

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=3da2836d321d4cb12a9ec56d0cfbaa77c048e9cd&newProject=try&newRevision=9278d43742c5d4853fadc12455ec13d40057a9d3

Comment 5

[Mike Conley (:mconley) (:⚙️)]

I spoke to Mardak and k88hudson about this today. I'm recording my thoughts here:

In summary, it sounds like what's being detected here isn't a regression in the normal sense. The about:home page acts differently now for the first few runs, and that's what the test is detecting. What I suggest is:

1. We pref the "extended triplets" feature off for the startup_about_home_paint Talos test
2. We introduce a new, separate Talos test that always shows the "extended triplets"
3. We introduce another separate Talos test that shows the "about:welcome" welcome page.

Comment 6

[Ed Lee :Mardak]

Attachment: Bug 1572682 - Switch performance tests to measure common case non-triplets new tab page

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50e71b06d6a7
Switch performance tests to measure common case non-triplets new tab page r=k88hudson

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/50e71b06d6a7

Comment 9

[Florin Strugariu [:Bebe]]

== Change summary for alert #22858 (as of Tue, 27 Aug 2019 01:55:10 GMT) ==

Improvements:

2% startup_about_home_paint linux64-shippable opt e10s stylo 752.21 -> 737.08

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22858