1466472 - 2.63 - 3.51% about_preferences_basic (linux64-qr, osx-10-10, windows10-64) regression on push 0dc7bfc7b0c1de8d390cea05f0248fbadeb4d5fc (Thu May 31 2018)

Status: RESOLVED WONTFIX

(Firefox :: New Tab Page, defect, P1)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=85c9f9c900314b7586d997bbedd67ccd8c8fad70&tochange=0dc7bfc7b0c1de8d390cea05f0248fbadeb4d5fc

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  4%  about_preferences_basic osx-10-10 opt e10s stylo     215.45 -> 223.02
  3%  about_preferences_basic windows10-64 pgo e10s stylo  130.82 -> 135.07
  3%  about_preferences_basic linux64-qr opt e10s stylo    153.48 -> 157.53


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=13592

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Here are the Gecko profiles:

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FLzqmWUxWReaRhStP05KruQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_about_preferences_basic.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FcDHqR_UiRLyXHa4nBP8cHg%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_about_preferences_basic.zip

Comment 2

[Ed Lee :Mardak]

The subtests view shows that the regression is all from loading the last page 9, about:preferences#home:

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=4ae4cc41c1ad65a6a9735bff436ab97cc1dfa5ee&newProject=autoland&newRevision=0dc7bfc7b0c1de8d390cea05f0248fbadeb4d5fc&originalSignature=11fb857eb64fb6cf53dace0636b89d06470e7108&newSignature=11fb857eb64fb6cf53dace0636b89d06470e7108&framework=1

https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/about-preferences/about_preferences_basic.manifest#9

So checking page_8_pagecycle_1 perf-html profiles:

Before: https://perfht.ml/2LnvhsC
After: https://perfht.ml/2LmU0xu

Looks like with the new "home-pane-loaded" notification, activity-stream code runs sooner -- while the page is loading. Whereas before, it happened to wait until search and sync are initialized before running. The activity stream code is taking the same amount of time before/after.

Notably, looking at the time of the first Navigation [end] Load marker
Before: 123ms
After: 151ms

And the time of last Paint [end] Rasterize marker
Before: 212ms
After: 166ms

rwood, I believe we want to accept this "load regression" / "paint improvement" and potentially update the pageloader test to (additionally?) measure when the page is more completely loaded. The main content of about:preferences#home is what gets loaded by activity stream, and from this one profile, it's being displayed 46ms sooner than before although the initial load for the rest of the page is delayed by 28ms. In particular, activity stream adding in preferences happened to bypass the measured load time before because it only added content after a later event, but now pageloader will more correctly measure any impacts of activity stream preference changes.

Comment 3

[Robert Wood [:rwood]]

Hi Ed,

Thanks for the info! From my point of view it's fine just accept this new value as a new test baseline. However it's more a call for the test owner, on the wiki [1] the owner of this test is listed as :jaws, so I am setting a ni? for :jaws.

From a talos development point of view: talos is not able to record multiple measurements per pageload - so we won't be able to add an additional measurement for this about:preferences test. The test currently measures first-non-blank-paint, but if you and :jaws want the test to measure something else instead then I could update Talos accordingly (however it has to be an accessible event that talos can wait for to measure).

[1] https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#about-preferences

Comment 4

[Ionuț Goldan [:igoldan]]

:jaws We remind you that we need your feedback for reaching a resolution here.

Comment 5

[Jared Wein [:jaws] (please needinfo? me)]

Yes, it is fine to accept this as the new value for the test baseline.

Apologies for not responding sooner.