Closed Bug 1544356 Opened 6 years ago Closed 5 years ago

4 - 281% raptor-tp6 regressions on all desktop platforms from push d55785a2bf02f8015aa0f1033f590342eea0510c (Thu April 11 2019)

Categories

(Testing :: Raptor, defect, P1)

Product:
Testing
Component:
Raptor
Type:
defect

Tracking

(firefox68 fixed)

Status:
VERIFIED FIXED
Milestone:
mozilla68
Tracking Flags:
Tracking Status
firefox68 --- fixed

People

(Reporter: igoldan, Assigned: Bebe)

References

(Regression)

Details

Attachments

(1 file)

Bug 1544356 4 - 281% raptor-tp6 regression
47 bytes, text/x-phabricator-request
Details | Review

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=20b05a5df3303e726d5e74ab0eb80bb5bd7478fd&tochange=d55785a2bf02f8015aa0f1033f590342eea0510c

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

281% raptor-tp6-google-firefox loadtime windows10-64-shippable-qr opt 124.42 -> 473.79
263% raptor-tp6-google-firefox loadtime windows10-64-shippable opt 128.25 -> 466.08
259% raptor-tp6-google-firefox loadtime windows7-32-shippable opt 129.46 -> 464.75
246% raptor-tp6-google-firefox loadtime linux64-shippable opt 136.83 -> 473.83
245% raptor-tp6-google-firefox loadtime linux64-shippable-qr opt 139.71 -> 481.46
197% raptor-tp6-amazon-firefox loadtime windows7-32-shippable opt 324.42 -> 963.00
195% raptor-tp6-amazon-firefox loadtime windows10-64-shippable opt 325.46 -> 958.88
184% raptor-tp6-amazon-firefox loadtime windows10-64-shippable-qr opt 334.42 -> 948.83
127% raptor-tp6-google-firefox windows10-64-shippable-qr opt 119.35 -> 270.72
124% raptor-tp6-google-firefox windows10-64-shippable opt 120.85 -> 270.92
123% raptor-tp6-google-firefox windows7-32-shippable opt 121.83 -> 271.98
115% raptor-tp6-google-firefox linux64-shippable opt 126.80 -> 272.41
110% raptor-tp6-amazon-firefox loadtime linux64-shippable-qr opt 479.38 -> 1,008.25
110% raptor-tp6-google-firefox linux64-shippable-qr opt 132.72 -> 279.05
110% raptor-tp6-amazon-firefox loadtime linux64-shippable opt 464.83 -> 975.58
102% raptor-tp6-facebook-firefox loadtime windows7-32-shippable opt 325.96 -> 658.29
95% raptor-tp6-facebook-firefox loadtime windows10-64-shippable opt 328.44 -> 640.46
84% raptor-tp6-google-firefox fcp windows10-64-shippable-qr opt 128.25 -> 236.08
83% raptor-tp6-google-firefox fcp windows7-32-shippable opt 130.83 -> 239.54
83% raptor-tp6-google-firefox fcp windows10-64-shippable opt 130.08 -> 237.83
77% raptor-tp6-google-firefox fcp linux64-shippable opt 134.17 -> 237.46
72% raptor-tp6-google-firefox fcp linux64-shippable-qr opt 142.42 -> 244.83
71% raptor-tp6-amazon-firefox windows10-64-shippable opt 276.26 -> 472.10
71% raptor-tp6-amazon-firefox windows7-32-shippable opt 276.62 -> 472.16
70% raptor-tp6-amazon-firefox windows10-64-shippable-qr opt 278.34 -> 472.51
59% raptor-tp6-facebook-firefox windows7-32-shippable opt 253.98 -> 403.93
59% raptor-tp6-facebook-firefox windows10-64-shippable opt 254.61 -> 404.06
56% raptor-tp6-amazon-firefox linux64-shippable opt 299.48 -> 467.03
55% raptor-tp6-amazon-firefox linux64-shippable-qr opt 317.71 -> 492.71
54% raptor-tp6-amazon-firefox loadtime osx-10-10-shippable opt 926.54 -> 1,422.54
53% raptor-tp6-google-firefox osx-10-10-shippable opt 224.17 -> 343.87
53% raptor-tp6-google-firefox loadtime osx-10-10-shippable opt 250.96 -> 383.83
50% raptor-tp6-microsoft-firefox linux64-shippable opt 638.85 -> 959.57
49% raptor-tp6-microsoft-firefox windows7-32-shippable opt 685.40 -> 1,018.85
49% raptor-tp6-microsoft-firefox osx-10-10-shippable opt 1,733.30 -> 2,575.67
47% raptor-tp6-microsoft-firefox linux64-shippable-qr opt 687.36 -> 1,012.70
46% raptor-tp6-microsoft-firefox windows10-64-shippable opt 693.53 -> 1,014.36
46% raptor-tp6-youtube-firefox linux64-shippable opt 282.31 -> 412.55
44% raptor-tp6-facebook-firefox linux64-shippable opt 245.82 -> 355.06
44% raptor-tp6-youtube-firefox windows10-64-shippable opt 304.20 -> 439.31
44% raptor-tp6-amazon-firefox fcp windows10-64-shippable-qr opt 255.42 -> 368.58
44% raptor-tp6-amazon-firefox fcp linux64-shippable opt 249.67 -> 359.58
44% raptor-tp6-facebook-firefox windows10-64-shippable-qr opt 263.47 -> 378.49
43% raptor-tp6-amazon-firefox fcp windows10-64-shippable opt 257.08 -> 368.50
43% raptor-tp6-google-firefox fcp osx-10-10-shippable opt 244.31 -> 350.08
43% raptor-tp6-amazon-firefox fcp windows7-32-shippable opt 258.46 -> 369.12
42% raptor-tp6-amazon-firefox fcp linux64-shippable-qr opt 268.88 -> 380.54
41% raptor-tp6-microsoft-firefox windows10-64-shippable-qr opt 713.95 -> 1,005.70
40% raptor-tp6-facebook-firefox linux64-shippable-qr opt 267.37 -> 374.08
40% raptor-tp6-facebook-firefox fcp windows10-64-shippable opt 286.33 -> 399.92
39% raptor-tp6-youtube-firefox windows10-64-shippable-qr opt 308.76 -> 429.21
38% raptor-tp6-facebook-firefox fcp windows7-32-shippable opt 284.71 -> 394.00
38% raptor-tp6-youtube-firefox windows7-32-shippable opt 310.24 -> 427.42
37% raptor-tp6-youtube-firefox linux64-shippable-qr opt 312.71 -> 428.09
36% raptor-tp6-youtube-firefox fcp linux64-shippable opt 266.71 -> 363.79
36% raptor-tp6-facebook-firefox fcp windows10-64-shippable-qr opt 293.21 -> 399.62
35% raptor-tp6-facebook-firefox fcp linux64-shippable opt 277.17 -> 375.50
34% raptor-tp6-sheets-firefox windows10-64-shippable-qr opt 270.47 -> 363.27
34% raptor-tp6-youtube-firefox fcp windows10-64-shippable opt 296.21 -> 395.96
34% raptor-tp6-sheets-firefox windows10-64-shippable opt 269.13 -> 359.72
33% raptor-tp6-youtube-firefox fcp windows7-32-shippable opt 296.12 -> 394.00
33% raptor-tp6-sheets-firefox windows7-32-shippable opt 269.85 -> 358.97
33% raptor-tp6-youtube-firefox fcp linux64-shippable-qr opt 286.21 -> 379.92
32% raptor-tp6-facebook-firefox fcp linux64-shippable-qr opt 300.75 -> 397.25
32% raptor-tp6-youtube-firefox fcp windows10-64-shippable-qr opt 295.88 -> 390.71
31% raptor-tp6-sheets-firefox linux64-shippable opt 250.46 -> 328.34
30% raptor-tp6-youtube-firefox loadtime windows10-64-shippable-qr opt 560.67 -> 731.08
30% raptor-tp6-sheets-firefox linux64-shippable-qr opt 266.32 -> 346.16
28% raptor-tp6-youtube-firefox loadtime linux64-shippable opt 550.50 -> 706.58
28% raptor-tp6-sheets-firefox osx-10-10-shippable opt 661.74 -> 849.29
27% raptor-tp6-imgur-firefox windows10-64-ux opt 462.21 -> 585.91
26% raptor-tp6-imgur-firefox osx-10-10-shippable opt 606.02 -> 764.84
26% raptor-tp6-imgur-firefox windows7-32-shippable opt 254.21 -> 320.69
24% raptor-tp6-imgur-firefox windows10-64-shippable opt 256.68 -> 319.26
21% raptor-tp6-docs-firefox osx-10-10-shippable opt 1,146.53 -> 1,388.04
20% raptor-tp6-imgur-firefox windows10-64-shippable-qr opt 260.33 -> 313.01
19% raptor-tp6-amazon-firefox osx-10-10-shippable opt 753.34 -> 893.65
17% raptor-tp6-youtube-firefox loadtime linux64-shippable-qr opt 621.96 -> 727.12
15% raptor-tp6-facebook-firefox osx-10-10-shippable opt 700.26 -> 805.50
12% raptor-tp6-amazon-firefox fcp osx-10-10-shippable opt 661.31 -> 740.83
8% raptor-tp6-docs-firefox linux64-shippable opt 767.26 -> 830.96
8% raptor-tp6-docs-firefox linux64-shippable-qr opt 820.15 -> 882.10
4% raptor-tp6-slides-firefox linux64-shippable-qr opt 736.41 -> 765.87

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=20407

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Florin, where these results expected?

Flags: needinfo?(fstrugariu)

yes this is caused by a framework change:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d55785a2bf02f8015aa0f1033f590342eea0510c&group_state=expanded

Flags: needinfo?(fstrugariu)

Bug 1519827 would explain regressions to geomean of the affected sites (Google Docs, Google Sheets, Google Slides, Imgur, Microsoft) and bug 1509835 would explain a regression of Amazon on Chromium.

I see some significant regressions on other sites though, such as Google, Amazon (Firefox), and Facebook. Could we investigate further? Perhaps there are a few alerts erroneuously combined under a single summary.

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #3)

I see some significant regressions on other sites though, such as Google, Amazon (Firefox), and Facebook. Could we investigate further? Perhaps there are a few alerts erroneuously combined under a single summary.

I'll do some retriggers/backfills on them.

This is because we are updating the full tp6-1 suite to mitmproxy 4.0.4

Try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9f0d85cc0c735bf86b9cab7d9b9e4fe18bb1e777

Assignee: nobody → fstrugariu

:bebe, if we are taking a hit from mitmproxy 4.0.4 then this issue can be closed (?)

Flags: needinfo?(fstrugariu)
Priority: -- → P1

:rwood we made a mistake when changing this and converted the full file to mitmproxy 4.0.4 and we take a unnecessary hit
this patch should fix this

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a2b6a9459921f404cbe9a64417d48c4d2e1fa1d

Flags: needinfo?(fstrugariu)
Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/859f2328e7aa
4 - 281% raptor-tp6 regression r=davehunt

https://hg.mozilla.org/mozilla-central/rev/859f2328e7aa

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox68: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

== Change summary for alert #20699 (as of Mon, 29 Apr 2019 18:55:10 GMT) ==

Improvements:

72% raptor-tp6-google-firefox loadtime windows10-64-shippable-qr opt 464.00 -> 130.38
72% raptor-tp6-google-firefox loadtime windows10-64-shippable opt 470.29 -> 134.04
71% raptor-tp6-google-firefox loadtime windows7-32-shippable opt 474.29 -> 136.08
71% raptor-tp6-google-firefox loadtime linux64-shippable-qr opt 484.96 -> 141.29
70% raptor-tp6-google-firefox loadtime linux64-shippable opt 476.83 -> 141.42
55% raptor-tp6-google-firefox windows10-64-shippable-qr opt 272.16 -> 123.83
54% raptor-tp6-google-firefox windows10-64-shippable opt 274.66 -> 125.97
54% raptor-tp6-google-firefox linux64-shippable-qr opt 280.99 -> 129.95
53% raptor-tp6-google-firefox windows7-32-shippable opt 276.12 -> 128.41
52% raptor-tp6-google-firefox linux64-shippable opt 274.03 -> 131.18
45% raptor-tp6-google-firefox fcp linux64-shippable-qr opt 248.79 -> 136.46
44% raptor-tp6-facebook-firefox loadtime windows7-32-shippable opt 570.79 -> 320.33
44% raptor-tp6-google-firefox fcp windows10-64-shippable opt 240.71 -> 135.29
44% raptor-tp6-google-firefox fcp windows10-64-shippable-qr opt 239.00 -> 134.71
43% raptor-tp6-google-firefox fcp windows7-32-shippable opt 244.00 -> 138.67
42% raptor-tp6-google-firefox fcp linux64-shippable opt 236.17 -> 136.25
39% raptor-tp6-facebook-firefox loadtime linux64-shippable opt 509.02 -> 309.58
38% raptor-tp6-facebook-firefox loadtime linux64-shippable-qr opt 528.71 -> 328.08
35% raptor-tp6-facebook-firefox windows7-32-shippable opt 385.40 -> 250.09
35% raptor-tp6-google-firefox osx-10-10-shippable opt 346.22 -> 225.35
34% raptor-tp6-facebook-firefox windows10-64-shippable opt 391.04 -> 257.31
34% raptor-tp6-google-firefox fcp osx-10-10-shippable opt 351.00 -> 231.54
33% raptor-tp6-google-firefox loadtime osx-10-10-shippable opt 387.92 -> 259.08
33% raptor-tp6-facebook-firefox linux64-shippable opt 359.97 -> 242.21
33% raptor-tp6-facebook-firefox windows10-64-shippable-qr opt 392.85 -> 264.62
31% raptor-tp6-facebook-firefox linux64-shippable-qr opt 374.81 -> 257.40
29% raptor-tp6-youtube-firefox windows7-32-shippable opt 434.00 -> 308.11
28% raptor-tp6-youtube-firefox linux64-shippable opt 409.73 -> 293.58
28% raptor-tp6-youtube-firefox windows10-64-shippable opt 434.87 -> 314.84
28% raptor-tp6-facebook-firefox fcp windows7-32-shippable opt 387.50 -> 280.62
27% raptor-tp6-youtube-firefox linux64-shippable-qr opt 424.50 -> 309.99
27% raptor-tp6-facebook-firefox fcp windows10-64-shippable opt 393.46 -> 287.54
27% raptor-tp6-facebook-firefox fcp linux64-shippable opt 370.29 -> 271.12
27% raptor-tp6-youtube-firefox windows10-64-shippable-qr opt 437.14 -> 320.27
26% raptor-tp6-facebook-firefox fcp windows10-64-shippable-qr opt 400.12 -> 295.17
26% raptor-tp6-facebook-firefox fcp linux64-shippable-qr opt 392.71 -> 290.54
25% raptor-tp6-youtube-firefox fcp linux64-shippable opt 361.17 -> 272.21
25% raptor-tp6-youtube-firefox fcp linux64-shippable-qr opt 375.33 -> 283.33
24% raptor-tp6-youtube-firefox fcp windows7-32-shippable opt 400.25 -> 303.79
22% raptor-tp6-youtube-firefox fcp windows10-64-shippable-qr opt 396.79 -> 307.79
22% raptor-tp6-youtube-firefox fcp windows10-64-shippable opt 399.54 -> 311.17
21% raptor-tp6-youtube-firefox loadtime windows7-32-shippable opt 721.67 -> 570.12
20% raptor-tp6-youtube-firefox loadtime windows10-64-shippable opt 726.04 -> 582.71
16% raptor-tp6-youtube-firefox loadtime windows10-64-shippable-qr opt 724.02 -> 605.50
16% raptor-tp6-youtube-firefox loadtime linux64-shippable-qr opt 718.00 -> 601.50
16% raptor-tp6-youtube-firefox loadtime linux64-shippable opt 693.60 -> 582.67
13% raptor-tp6-facebook-firefox osx-10-10-shippable opt 827.39 -> 722.03
12% raptor-tp6-facebook-firefox fcp osx-10-10-shippable opt 752.56 -> 664.96
7% raptor-tp6-facebook-firefox loadtime osx-10-10-shippable opt 1,617.50 -> 1,509.04

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=20699

Status: RESOLVED → VERIFIED
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: