1607750 - 1.84 - 2.5% raptor-tp6-microsoft-firefox-cold fcp (windows10-64-shippable-qr) regression on push 80f5c59d5df430ecff942865c54c7bb3c8ee4257 (Thu January 2 2020)

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=80f5c59d5df430ecff942865c54c7bb3c8ee4257

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

3% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable-qr opt 521.79 -> 534.83
2% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable-qr opt 526.38 -> 536.08

Improvements:

66% raptor-tp6-yahoo-mail-firefox-cold loadtime windows7-32-shippable opt 2,719.17 -> 919.67
33% raptor-tp6-yahoo-mail-firefox-cold loadtime windows10-64-shippable opt 1,361.50 -> 911.00
24% raptor-tp6-yahoo-mail-firefox-cold windows7-32-shippable opt 651.71 -> 497.76
16% raptor-tp6-tumblr-firefox-cold loadtime windows7-32-shippable opt 1,933.17 -> 1,622.42
10% raptor-tp6-pinterest-firefox-cold loadtime windows7-32-shippable opt 1,998.58 -> 1,791.25
10% raptor-tp6-yahoo-mail-firefox-cold windows10-64-shippable opt 544.76 -> 488.64
8% raptor-tp6m-google-maps-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 1,261.07 -> 1,158.81
8% raptor-tp6-tumblr-firefox-cold loadtime windows10-64-shippable opt 1,785.12 -> 1,643.25
8% raptor-tp6m-google-maps-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 1,207.83 -> 1,112.25
8% raptor-tp6m-google-maps-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 1,667.06 -> 1,542.00
7% raptor-tp6-pinterest-firefox-cold loadtime windows10-64-shippable opt 1,929.58 -> 1,795.50
7% raptor-tp6m-google-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 1,183.83 -> 1,103.67
7% raptor-tp6m-google-maps-geckoview-cold fcp android-hw-p2-8-0-android-aarch64 pgo 573.58 -> 534.92
7% raptor-tp6m-bing-restaurants-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 577.09 -> 539.52
7% raptor-tp6m-bing-restaurants-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 619.00 -> 578.75
6% raptor-tp6m-google-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 563.62 -> 527.60
6% raptor-tp6m-bing-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 505.38 -> 473.33
6% raptor-tp6m-bing-geckoview-cold android-hw-g5-7-0-arm7-api-16 pgo 484.71 -> 455.45
6% raptor-tp6m-bing-restaurants-geckoview-cold fcp android-hw-g5-7-0-arm7-api-16 pgo 704.19 -> 665.25
5% raptor-tp6m-instagram-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 3,051.15 -> 2,896.00

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24619

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

From my initial investigation, here, I've filed bug 1608241 (mitmproxy exits with error code 572 but Raptor test still passes), I found an additional case[0] where: a) mitmproxy exits with error code 572, but b) the screenshots appear complete[1].

Using the greatest "improvement" job of raptor-tp6-yahoo-mail-firefox-cold loadtime windows10-64-shippable opt[2], perf-wise, I also found that this mitmproxy error predates[3] my commit.

I think this means we can't solely implicate these mitmproxy errors, nor -- for this particular performance regression -- rely on "expected" vs. actual screenshots, but we should definitely:

A) back out bug 1602657 while we look further
B) dive into bug 1608241 and do our best to remove it from the equation/problem set
C) investigate the discrepancy in the screenshots - perhaps doing B) will help with this, but I have no idea

D) only after we've vetted and gotten a handle on these issues should we re-introduce this patch.

--
[0] https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284107929&repo=try&lineNumber=4153
[1] https://firefoxci.taskcluster-artifacts.net/IPeC5zuPQh2H0We1P56mzg/0/public/test_info/screenshots.html
[2] https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2134904,1,10&series=mozilla-inbound,2137384,1,10&timerange=1209600&zoom=1577961780483,1578074182414,2.3977559535925015,1522.5750305312438
[3] https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283255097&repo=autoland&lineNumber=4293

Comment 2

[Marian Raiciof [:marauder]]

The backout from https://bugzilla.mozilla.org/show_bug.cgi?id=1602657#c7
shows improvements in the graph, the regressions were fixed:

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=325da1027&selected=2134557,1022316415&series=autoland,2134557,1,10&series=mozilla-inbound,2137022,1,10&timerange=1209600&zoom=1577704835000,1578897531000,195.66666666666666,303.3333333333333

Comment 3

[Marian Raiciof [:marauder]]

I changed the alert state to backedout:
https://treeherder.mozilla.org/perf.html#/alerts?id=24619

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Do we need this backout on Beta73 also?

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #4)

Do we need this backout on Beta73 also?

It's a "CI-only" patch, so really shouldn't affect end-users, and I'd think can ride the train when it's next appropriate, but I'm not 100% clear (not sure if bug 1605237/any other uplifts come into play, here, too).

Tarek, mind helping clarify/clear this up, if you would? Thanks!

Comment 6

[Tarek Ziadé (:tarek)]

since we run the perf test on the beta branch as well, we should uplift this backout there. Thanks

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Backed out from Beta.
https://hg.mozilla.org/releases/mozilla-beta/rev/d2dfae9a06bf

Comment 8

[Florin Strugariu [:Bebe]]

== Change summary for alert #24733 (as of Wed, 22 Jan 2020 01:01:18 GMT) ==

Regressions:

9% raptor-tp6-google-firefox-cold fcp windows10-64-shippable-qr opt 319.35 -> 346.75
7% raptor-tp6-twitch-firefox-cold fcp linux64-shippable opt 75.31 -> 80.33

Improvements:

7% raptor-tp6-netflix-firefox-cold loadtime windows10-64-shippable-qr opt 1,868.75 -> 1,735.33
2% raptor-tp6-sheets-firefox-cold fcp windows7-32-shippable opt 790.81 -> 771.42
2% raptor-tp6-sheets-firefox-cold windows7-32-shippable opt 911.14 -> 892.60
2% raptor-tp6-sheets-firefox-cold loadtime windows7-32-shippable opt 1,620.50 -> 1,587.58

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24733