1635053 - 2.42 - 4.72% *-cold-mozproxy-replay (android-hw-p2-8-0-android-aarch64, linux64-shippable, macosx1014-64-shippable, windows10-64-shippable,) regression on push 1fd9c358e8934150fb43384d553f9ada79ec08dc (Wed April 29 2020)

Status: RESOLVED WONTFIX

(Testing :: Performance, defect, P3)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a raptor performance regression from push 1fd9c358e8934150fb43384d553f9ada79ec08dc. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% raptor-tp6m-cnn-ampstories-geckoview-cold-mozproxy-replay android-hw-p2-8-0-android-aarch64 pgo 98.83 -> 94.17
4% raptor-tp6-bing-firefox-cold-mozproxy-replay windows7-32-shippable opt 51.00 -> 49.00
4% raptor-tp6-bing-firefox-cold-mozproxy-replay windows10-64-shippable opt 51.00 -> 49.08
4% raptor-tp6-bing-firefox-cold-mozproxy-replay windows10-64-shippable-qr opt 51.00 -> 49.08
3% raptor-tp6-amazon-firefox-mitm5-cold-mozproxy-replay macosx1014-64-shippable opt 65.25 -> 63.00
3% raptor-tp6-yahoo-mail-firefox-cold-mozproxy-replay linux64-shippable opt 79.67 -> 77.17
3% raptor-tp6-facebook-redesign-firefox-cold-mozproxy-replay macosx1014-64-shippable opt 84.33 -> 82.08
3% raptor-tp6-bing-firefox-cold-mozproxy-replay macosx1014-64-shippable opt 51.88 -> 50.50
2% raptor-tp6-yahoo-mail-firefox-cold-mozproxy-replay linux64-shippable opt 79.33 -> 77.42

Improvements:

7% raptor-tp6m-wikipedia-geckoview-cold loadtime android-hw-g5-7-0-arm7-api-16 pgo 1,024.08 -> 948.33
7% raptor-tp6m-bing-geckoview-cold-mozproxy-replay android-hw-p2-8-0-android-aarch64 pgo 79.62 -> 85.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is nearly all mozproxy-relay. Dave, I remember we chatted last week about those tests, and that you wanted to talk to Florin about those tests. What was the outcome? How actionable are those results? AFAIR we should ignore those?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The replay confidence has dropped, which could be expected if tests are now running to completion where they weren't previously. Either way, we can accept this regression and address improving the confidence separately.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Dave. I assume WFM is the right way to close such a bug?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We usually use WONTFIX to accept performance changes.