Closed Bug 1649100 Opened 2 years ago Closed 2 years ago

3.63% Browsertime vismet regression on push 8aff011594bfdeb0a7d364ebf4b18ba755477319 (Tue June 23 2020)

Categories

(Testing :: Performance, defect, P1)

79 Branch
defect

Tracking

(firefox79 wontfix)

RESOLVED WONTFIX
Tracking Status
firefox79 --- wontfix

People

(Reporter: Bebe, Unassigned)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Perfherder has detected a browsertime performance regression from push 8aff011594bfdeb0a7d364ebf4b18ba755477319. As author of one of the patches included in that push, we need your help to address this regression.

== Change summary for alert #26322 (as of Wed, 24 Jun 2020 13:16:04 GMT) ==

Regressions:

41% reddit ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,959.08 -> 4,160.17
35% youtube-watch ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,229.46 -> 1,655.25
30% reddit ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt cold 1,534.67 -> 1,994.83
21% instagram ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 812.58 -> 983.17
11% ebay-kleinanzeigen-search ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,945.96 -> 2,162.17
10% reddit loadtime android-hw-g5-7-0-arm7-api-16-shippable opt cold 4,779.81 -> 5,242.58
7% youtube ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,042.71 -> 1,117.25
6% reddit LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 6,293.33 -> 6,701.58
6% reddit SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 5,399.50 -> 5,738.00
5% bing-search-restaurants ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 670.88 -> 707.25
5% youtube-watch LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,506.67 -> 2,622.08
4% bing-search-restaurants FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 636.67 -> 661.33
3% imdb LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 10,301.67 -> 10,634.83

Improvements:

65% cnn videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 735.00
65% bbc videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 735.42
65% jianshu videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 736.92
65% microsoft-support videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 736.92
65% amazon-search videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 737.08
65% allrecipes videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 738.00
64% ebay-kleinanzeigen-search videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 738.58
64% google-search-restaurants videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 738.67
64% web-de videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 738.83
64% imdb videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 739.00
64% reddit videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 739.08
64% cnn-ampstories videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 739.25
64% facebook-cristiano videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 739.25
64% booking videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 739.58
64% ebay-kleinanzeigen videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 740.08
64% bing videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 740.25
64% google videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 740.50
64% instagram videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 740.50
64% youtube videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 741.25
64% youtube-watch videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,076.67 -> 740.33
64% amazon videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,073.33 -> 739.67
64% stackoverflow videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 742.58
64% wikipedia videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 743.58
64% espn videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 743.83
64% google-maps videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 744.25
64% bing-search-restaurants videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,076.67 -> 744.42
64% facebook videoRecordingStart android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,080.00 -> 748.17
21% allrecipes SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 5,536.92 -> 4,388.00
18% allrecipes SpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt cold 2,439.71 -> 1,989.75
10% bing ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,172.21 -> 1,058.08
10% google-maps LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 7,056.67 -> 6,380.00
7% facebook-cristiano videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,903.58
7% bing-search-restaurants videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,904.00
7% web-de videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,904.42
7% jianshu videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,904.92
7% microsoft-support videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,904.92
7% amazon-search videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,905.17
7% youtube videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,905.25
7% ebay-kleinanzeigen-search videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,905.58
7% instagram videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,905.83
7% youtube-watch videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,905.75
7% google videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.00
7% allrecipes videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.17
7% facebook videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.33
7% cnn-ampstories videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.50
7% ebay-kleinanzeigen videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.50
7% google-maps videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.50
7% imdb videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.67
7% reddit videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.58
7% cnn videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.75
7% google-search-restaurants videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.75
7% amazon videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.92
7% stackoverflow videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,906.92
7% booking videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,036.67 -> 1,904.17
7% espn videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,907.33
6% bbc videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,907.67
6% bing videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,907.67
6% wikipedia videoRecordingStart android-hw-p2-8-0-android-aarch64-shippable opt cold 2,040.00 -> 1,907.75

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26322

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

We can resolve this as WONTFIX since these are caused by a temporary tooling change (a known shift to our baseline).

Severity: -- → S3
Status: NEW → RESOLVED
Closed: 2 years ago
Priority: -- → P1
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.