1571624 - 5.07 - 21.57% Images (linux64-shippable-qr, windows10-64-shippable-qr) regression on push 37229cef2cc79d44470afc9e04016bac8ddd0ae8 (Fri August 2 2019)

Status: RESOLVED FIXED

(Firefox :: New Tab Page, defect, P2)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=37229cef2cc79d44470afc9e04016bac8ddd0ae8

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

22% Images linux64-shippable-qr opt stylo tp6 8,529,808.50 -> 10,369,912.98
19% Images windows10-64-shippable-qr opt 7,496,049.95 -> 8,951,452.91
5% Images linux64-shippable-qr opt 5,927,519.55 -> 6,227,785.36

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=22270

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Andrei Oprea [:andreio]]

Can we please get the profile information from those tests?

Comment 2

[Ionuț Goldan [:igoldan]]

(In reply to Andrei Oprea [:andreio] from comment #1)

Can we please get the profile information from those tests?

Unfortunately, we don't provide Gecko profiles for AWSY perf tests.

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Andrei Oprea [:andreio] from comment #1)

Can we please get the profile information from those tests?

If it helps in any way, Bug 1570745 was backed out once here

Comment 4

[Ed Lee :Mardak]

The runs do clearly show it is related to the export bug 1570745:

https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=0928c6245907b29a632be53031c0f4ea358151ee&searchStr=%28sy%2C10-64%2Cqr&fromchange=ee3b704f5009d1234dba2bb36914d11230724926

7.6 before initial landing
8.9 after initial landing
7.5 after backout
9.2 after relanding

Similar to bug 1572682, this is because we are now showing triplets images on about:home/newtab for new users. Ideally we have a new reference "with triplets" and "without triplets" where the former would be the new baseline and latter should match pre-regression levels.

The "without triplets" can be done now by running talos with trailhead.firstrun.branches=join pref to match the common case behavior of users not seeing triplets. But it would still be good to know how performance changes for new users who will be seeing triplets.

Comment 5

[Ed Lee :Mardak]

Here's a try run setting branch to join for awsy:
https://hg.mozilla.org/try/rev/625100a6d1f8ac7a625279a70807509b25f6a3cb

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=3da2836d321d4cb12a9ec56d0cfbaa77c048e9cd&newProject=try&newRevision=8b1ac603f176788dad083b7184afd9f282ac8861&framework=4

Comment 6

[Ed Lee :Mardak]

Attachment: Bug 1571624 - Switch awsy tests to measure common case non-triplets new tab page

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6ddb703d8e53
Switch awsy tests to measure common case non-triplets new tab page r=erahm

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/6ddb703d8e53

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #22445 (as of Wed, 14 Aug 2019 11:01:51 GMT) ==

Improvements:

20% Images linux64-shippable-qr opt stylo tp6 10,303,742.84 -> 8,202,356.50
12% Images windows10-64-shippable-qr opt 8,384,749.27 -> 7,413,176.62
6% Images linux64-shippable opt 4,978,494.14 -> 4,668,896.24
5% Images macosx1014-64-shippable opt 4,806,900.02 -> 4,576,701.82
5% Images windows10-64-shippable opt 5,768,974.36 -> 5,505,433.90

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=22445

Comment 12

[Ed Lee :Mardak]

Probably not worthwhile uplifting the "fix" to beta/rc as it doesn't change the user facing behavior.