1600332 - 5.36 - 12.53% Images (linux64, linux64-qr, linux64-shippable, linux64-shippable-qr, macosx1014-64-shippable, windows10-64-qr, windows10-64-shippable-qr) regression on push 9c53909304379e3c88e886cb9a83ca88cb822e5b (Thu November 28 2019)

Status: RESOLVED FIXED

(Firefox :: Messaging System, defect, P1)

Description

[Marian Raiciof [:marauder]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=9c53909304379e3c88e886cb9a83ca88cb822e5b

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

13% Images macosx1014-64-shippable opt 4,600,328.71 -> 5,176,889.66
12% Images macosx1014-64-shippable opt 4,631,387.37 -> 5,181,332.19
10% Images windows10-64-shippable-qr opt 8,309,780.56 -> 9,163,397.26
10% Images windows10-64-qr opt 8,219,383.60 -> 9,031,182.95
9% Images linux64-shippable-qr opt 5,920,638.35 -> 6,469,098.58
9% Images linux64-qr opt 5,915,116.56 -> 6,432,164.98
8% Images linux64 opt stylo tp6 7,602,802.68 -> 8,188,803.10
8% Images linux64-shippable opt stylo tp6 7,584,408.54 -> 8,158,764.81
8% Images macosx1014-64-shippable opt stylo tp6 7,912,990.39 -> 8,508,590.01
6% Images linux64-shippable-qr opt stylo tp6 8,584,373.84 -> 9,134,802.54
6% Images linux64-qr opt stylo tp6 8,572,847.16 -> 9,087,364.01
5% Images windows10-64-qr opt stylo tp6 9,505,374.49 -> 10,015,261.20

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24177

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Punam Dahiya [:pdahiya]]

Here are my findings

With landing of 1592671, setting trailhead.firstrun.branches to 'join' no longer hides triplets on new tab for new users essentially muting the fix here https://bugzilla.mozilla.org/show_bug.cgi?id=1571624#c4

With 1592671, if branches pref is not set interrupt is set to default 'join' and triplets are set to default 'supercharge'
https://searchfox.org/mozilla-central/source/browser/components/newtab/lib/ASRouterPreferences.jsm#24

The "without triplets" perf measure can be done now by running talos with trailhead.firstrun.branches='join-none' pref to match the common case behavior of users not seeing triplets. Will submit patch to fix pref here https://searchfox.org/mozilla-central/source/testing/awsy/conf/prefs.json#13

Comment 2

[Punam Dahiya [:pdahiya]]

Attachment: Bug 1600332 - Fix awsy tests to measure common case non-triplets new tab page

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78a0700674d9
Fix awsy tests to measure common case non-triplets new tab page r=Mardak

Comment 4

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/78a0700674d9

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #24217 (as of Tue, 03 Dec 2019 06:27:39 GMT) ==

Improvements:

19% Images windows7-32 opt 6,125,069.85 -> 4,983,976.26
16% Images windows7-32-shippable opt 6,152,571.98 -> 5,142,714.55
16% Images macosx1014-64-shippable opt 5,388,984.94 -> 4,514,398.86
16% Images linux64-stylo-sequential opt 5,357,073.82 -> 4,522,447.88
15% Images linux64 opt 5,388,729.06 -> 4,560,986.72
14% Images linux64-shippable opt 5,271,458.92 -> 4,537,148.56
13% Images windows10-64-shippable opt 6,246,640.44 -> 5,422,743.04
13% Images windows10-64-shippable-qr opt 9,351,936.40 -> 8,137,988.42
13% Images windows10-64-qr opt 9,248,714.74 -> 8,053,931.32
12% Images linux64-shippable-qr opt 6,520,144.34 -> 5,740,840.78
11% Images windows10-64 opt 6,588,886.90 -> 5,834,435.06
11% Images linux64-qr opt 6,557,011.91 -> 5,810,198.23
11% Images windows7-32-shippable opt stylo tp6 8,029,614.90 -> 7,169,649.71
10% Images macosx1014-64-shippable opt stylo tp6 8,662,176.69 -> 7,794,276.07
10% Images linux64-shippable opt stylo tp6 8,278,254.42 -> 7,465,916.54
10% Images linux64 opt stylo tp6 8,291,135.48 -> 7,485,777.30
9% Images linux64-shippable-qr opt stylo tp6 9,287,009.42 -> 8,453,867.94
9% Images windows7-32 opt stylo tp6 7,911,887.69 -> 7,221,503.44
8% Images windows10-64-qr opt stylo tp6 10,132,012.49 -> 9,338,871.98
7% Images windows10-64-shippable-qr opt stylo tp6 10,189,280.59 -> 9,425,882.80
7% Images linux64-qr opt stylo tp6 9,170,742.89 -> 8,484,510.26
4% Images windows10-64 opt stylo tp6 8,866,382.96 -> 8,490,926.08

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24217

Comment 6

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/6343070913bc

Comment 7

[Marian Raiciof [:marauder]]

== Change summary for alert #24479 (as of Fri, 13 Dec 2019 11:49:56 GMT) ==

Improvements:

19% Images windows7-32-shippable opt 6,195,823.07 -> 5,033,292.13
16% Images linux64-shippable opt 5,470,240.41 -> 4,617,960.70
15% Images macosx1014-64-shippable opt 5,308,865.49 -> 4,505,594.93
13% Images windows10-64-shippable opt 6,286,388.48 -> 5,481,359.38
12% Images windows10-64-shippable-qr opt 9,316,876.58 -> 8,219,223.43
11% Images linux64-shippable-qr opt 6,552,291.21 -> 5,855,485.89
10% Images linux64-shippable opt stylo tp6 8,350,077.65 -> 7,524,139.76
10% Images macosx1014-64-shippable opt stylo tp6 8,695,537.24 -> 7,865,530.03
8% Images linux64-shippable-qr opt stylo tp6 9,257,312.56 -> 8,477,317.14
8% Images windows10-64-shippable-qr opt stylo tp6 10,210,267.86 -> 9,443,740.51

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24479