1666190 - 2.22 - 16.69% Similarity / Similarity2D (android-hw-g5-7-0-arm7-api-16-shippable, android-hw-p2-8-0-android-aarch64-shippable) regression on push 35e97623d658bd0f2b300c60c2dd3f69dfe31a2d (Sun September 13 2020)

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Perfherder has detected a browsertime performance regression from push 35e97623d658bd0f2b300c60c2dd3f69dfe31a2d. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

17% bbc Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.99 -> 0.82
16% bbc Similarity2D android-hw-g5-7-0-arm7-api-16-shippable opt cold 1.00 -> 0.84
12% bbc Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.94 -> 0.82
12% microsoft-support Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.97 -> 0.86
10% bing Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 1.00 -> 0.90
7% microsoft-support Similarity android-hw-p2-8-0-android-aarch64-shippable opt cold 0.96 -> 0.90
7% espn Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.93 -> 0.86
6% bbc Similarity android-hw-p2-8-0-android-aarch64-shippable opt cold 0.99 -> 0.93
6% youtube-watch Similarity android-hw-g5-7-0-arm7-api-16-shippable opt cold 0.79 -> 0.74
5% espn Similarity android-hw-p2-8-0-android-aarch64-shippable opt cold 0.94 -> 0.89
5% stackoverflow Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.92 -> 0.88
4% ebay-kleinanzeigen-search Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.98 -> 0.94
3% ebay-kleinanzeigen Similarity2D android-hw-p2-8-0-android-aarch64-shippable opt cold 0.94 -> 0.91
2% ebay-kleinanzeigen-search Similarity android-hw-p2-8-0-android-aarch64-shippable opt cold 0.97 -> 0.95
2% bbc Similarity android-hw-g5-7-0-arm7-api-16-shippable opt cold 1.00 -> 0.97

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Greg Mierzwinski [:sparky]]

Ah this is because the similarity metric now pools all of cold and warm pageloads together in its calculations. See bug 1666238.