1591864 - 2.36 - 51.7% raptor-tp6 on push 25b533bff4051e6a8177bbc83eed49ac460e109e (Wed October 23 2019)

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f748a3d2cdf108e9443fd15332efe477c7c398a9&tochange=25b533bff4051e6a8177bbc83eed49ac460e109e

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

52% raptor-tp6-twitch-firefox fcp windows10-64-shippable-qr opt 50.21 -> 76.17
48% raptor-tp6-twitch-firefox fcp windows10-64-shippable opt 51.42 -> 76.00
47% raptor-tp6-linkedin-firefox-cold fcp windows10-64-shippable-qr opt 229.42 -> 337.25
39% raptor-tp6-linkedin-firefox-cold fcp macosx1014-64-shippable opt 430.33 -> 599.50
38% raptor-tp6-linkedin-firefox-cold fcp windows7-32-shippable opt 261.00 -> 360.08
37% raptor-tp6-linkedin-firefox-cold fcp windows10-64-shippable opt 257.42 -> 352.42
34% raptor-tp6-twitch-firefox fcp linux64-shippable opt 55.04 -> 73.54
32% raptor-tp6-twitch-firefox fcp linux64-shippable-qr opt 55.10 -> 72.75
29% raptor-tp6-twitch-firefox windows10-64-shippable opt 140.32 -> 181.26
28% raptor-tp6-twitch-firefox windows7-32-shippable opt 137.92 -> 176.42
26% raptor-tp6-twitch-firefox fcp macosx1014-64-shippable opt 69.92 -> 88.38
25% raptor-tp6-twitch-firefox windows10-64-shippable-qr opt 139.47 -> 174.10
24% raptor-tp6-linkedin-firefox-cold windows10-64-shippable-qr opt 571.70 -> 710.17
24% raptor-tp6-linkedin-firefox-cold macosx1014-64-shippable opt 898.38 -> 1,111.76
23% raptor-tp6-twitch-firefox windows10-64-shippable-qr opt 139.19 -> 171.71
23% raptor-tp6-linkedin-firefox-cold windows10-64-shippable opt 593.39 -> 731.14
21% raptor-tp6-twitch-firefox linux64-shippable opt 144.41 -> 174.72
20% raptor-tp6-linkedin-firefox-cold windows7-32-shippable opt 598.79 -> 719.38
20% raptor-tp6-twitch-firefox linux64-shippable-qr opt 148.74 -> 177.89
12% raptor-tp6-outlook-firefox-cold fcp macosx1014-64-shippable opt 303.88 -> 339.58
10% raptor-tp6-pinterest-firefox-cold loadtime linux64-shippable opt 1,794.67 -> 1,966.92
10% raptor-tp6-pinterest-firefox-cold loadtime linux64-shippable-qr opt 1,841.92 -> 2,017.17
8% raptor-tp6-pinterest-firefox-cold loadtime macosx1014-64-shippable opt 2,926.83 -> 3,149.58
7% raptor-tp6-yahoo-mail-firefox-cold loadtime windows7-32-shippable opt 2,467.00 -> 2,639.08
7% raptor-tp6-pinterest-firefox-cold loadtime windows10-64-shippable-qr opt 1,714.04 -> 1,831.00
4% raptor-tp6-google-mail-firefox-cold fcp linux64-shippable-qr opt 200.67 -> 209.00
4% raptor-tp6-google-mail-firefox-cold fcp windows7-32-shippable opt 238.67 -> 247.17
3% raptor-tp6-google-mail-firefox-cold fcp windows10-64-shippable opt 237.38 -> 245.25
3% raptor-tp6-google-mail-firefox-cold fcp windows10-64-shippable-qr opt 192.58 -> 198.83
3% raptor-tp6-google-mail-firefox-cold windows7-32-shippable opt 261.05 -> 268.96
3% raptor-tp6-google-mail-firefox-cold windows10-64-shippable opt 261.95 -> 269.39
3% raptor-tp6-google-mail-firefox linux64-shippable-qr opt 266.34 -> 273.46
2% raptor-tp6-google-mail-firefox loadtime linux64-shippable-qr opt 436.46 -> 447.25
2% raptor-tp6-google-mail-firefox fcp linux64-shippable-qr opt 237.12 -> 242.92
2% raptor-tp6-google-mail-firefox loadtime linux64-shippable opt 412.90 -> 422.75
2% raptor-tp6-google-mail-firefox-cold windows10-64-shippable-qr opt 246.47 -> 252.28

Improvements:

74% raptor-tp6-outlook-firefox windows7-32-shippable opt 284.24 -> 73.72
67% raptor-tp6-outlook-firefox windows10-64-shippable opt 328.68 -> 108.31
65% raptor-tp6-outlook-firefox linux64-shippable-qr opt 311.40 -> 107.88
65% raptor-tp6-outlook-firefox windows10-64-shippable-qr opt 337.46 -> 119.57
58% raptor-tp6-outlook-firefox linux64-shippable opt 281.37 -> 119.10
51% raptor-tp6-outlook-firefox macosx1014-64-shippable opt 728.15 -> 360.36
49% raptor-tp6-outlook-firefox loadtime windows7-32-shippable opt 394.04 -> 199.17
48% raptor-tp6-outlook-firefox loadtime windows10-64-shippable opt 446.08 -> 233.25
45% raptor-tp6-outlook-firefox loadtime windows10-64-shippable-qr opt 457.42 -> 249.88
43% raptor-tp6-outlook-firefox loadtime linux64-shippable-qr opt 429.04 -> 246.12
40% raptor-tp6-outlook-firefox loadtime linux64-shippable opt 389.35 -> 234.29
37% raptor-tp6-outlook-firefox loadtime macosx1014-64-shippable opt 933.06 -> 589.46
11% raptor-tp6-instagram-firefox-cold fcp linux64-shippable opt 212.33 -> 188.83
10% raptor-tp6-linkedin-firefox loadtime windows10-64-shippable-qr opt 1,567.79 -> 1,413.62
10% raptor-tp6-outlook-firefox-cold fcp linux64-shippable opt 263.75 -> 238.25
9% raptor-tp6-reddit-firefox-cold fcp linux64-shippable opt 281.00 -> 255.58
9% raptor-tp6-bing-firefox-cold fcp linux64-shippable opt 263.83 -> 240.17
9% raptor-tp6-instagram-firefox-cold fcp linux64-shippable-qr opt 211.25 -> 192.92
9% raptor-tp6-outlook-firefox-cold fcp windows10-64-shippable opt 228.33 -> 208.92
8% raptor-tp6-outlook-firefox-cold linux64-shippable opt 342.68 -> 314.67
8% raptor-tp6-yahoo-mail-firefox-cold fcp linux64-shippable opt 312.00 -> 286.50
8% raptor-tp6-outlook-firefox-cold fcp windows10-64-shippable-qr opt 229.17 -> 210.58
8% raptor-tp6-bing-firefox-cold linux64-shippable opt 268.34 -> 246.95
8% raptor-tp6-linkedin-firefox loadtime windows10-64-shippable opt 1,531.56 -> 1,410.33
8% raptor-tp6-google-firefox-cold fcp linux64-shippable opt 325.67 -> 299.92
8% raptor-tp6-instagram-firefox-cold linux64-shippable opt 401.90 -> 370.32
8% raptor-tp6-instagram-firefox-cold fcp windows10-64-shippable opt 218.17 -> 201.17
8% raptor-tp6-outlook-firefox-cold windows10-64-shippable opt 280.25 -> 259.15
8% raptor-tp6-outlook-firefox-cold fcp windows7-32-shippable opt 227.08 -> 210.00
7% raptor-tp6-outlook-firefox-cold windows7-32-shippable opt 279.18 -> 259.67
7% raptor-tp6-tumblr-firefox loadtime windows7-32-shippable opt 662.83 -> 616.75
7% raptor-tp6-tumblr-firefox loadtime macosx1014-64-shippable opt 1,038.94 -> 967.88
7% raptor-tp6-bing-firefox-cold fcp windows10-64-shippable-qr opt 277.50 -> 258.75
7% raptor-tp6-outlook-firefox-cold windows10-64-shippable-qr opt 279.69 -> 260.92
7% raptor-tp6-bing-firefox-cold windows7-32-shippable opt 254.60 -> 237.59
7% raptor-tp6-bing-firefox-cold windows10-64-shippable-qr opt 254.72 -> 237.93
7% raptor-tp6-reddit-firefox-cold fcp windows10-64-shippable opt 313.17 -> 292.58
6% raptor-tp6-reddit-firefox-cold fcp linux64-shippable-qr opt 304.92 -> 285.25
6% raptor-tp6-google-firefox-cold linux64-shippable opt 426.59 -> 399.21
6% raptor-tp6-bing-firefox-cold windows10-64-shippable opt 253.06 -> 236.88
6% raptor-tp6-instagram-firefox-cold fcp windows7-32-shippable opt 219.08 -> 205.17
6% raptor-tp6-instagram-firefox-cold linux64-shippable-qr opt 398.70 -> 373.60
6% raptor-tp6-google-firefox-cold fcp linux64-shippable-qr opt 324.92 -> 304.50
6% raptor-tp6-bing-firefox-cold loadtime linux64-shippable opt 341.25 -> 320.17
6% raptor-tp6-reddit-firefox-cold fcp windows10-64-shippable-qr opt 296.50 -> 278.25
6% raptor-tp6-bing-firefox-cold fcp windows7-32-shippable opt 275.21 -> 258.42
6% raptor-tp6-instagram-firefox-cold fcp windows10-64-shippable-qr opt 220.71 -> 207.75
6% raptor-tp6-reddit-firefox-cold fcp windows7-32-shippable opt 316.92 -> 298.67
6% raptor-tp6-linkedin-firefox loadtime windows7-32-shippable opt 1,512.12 -> 1,425.12
6% raptor-tp6-reddit-firefox-cold linux64-shippable opt 741.56 -> 699.27
6% raptor-tp6-tumblr-firefox loadtime windows10-64-shippable-qr opt 665.15 -> 627.71
6% raptor-tp6-yahoo-mail-firefox-cold linux64-shippable opt 525.00 -> 495.89
5% raptor-tp6-bing-firefox-cold fcp windows10-64-shippable opt 272.71 -> 257.83
5% raptor-tp6-bing-firefox-cold loadtime windows10-64-shippable opt 322.17 -> 304.92
5% raptor-tp6-yahoo-mail-firefox-cold fcp windows10-64-shippable-qr opt 367.62 -> 348.25
5% raptor-tp6-google-firefox-cold fcp windows7-32-shippable opt 321.33 -> 304.42
5% raptor-tp6-outlook-firefox-cold linux64-shippable-qr opt 339.07 -> 321.25
5% raptor-tp6-instagram-firefox-cold windows10-64-shippable-qr opt 405.51 -> 384.22
5% raptor-tp6-bing-firefox-cold loadtime windows7-32-shippable opt 321.42 -> 304.83
5% raptor-tp6-yahoo-mail-firefox-cold fcp windows10-64-shippable opt 302.92 -> 287.42
5% raptor-tp6-google-firefox-cold fcp windows10-64-shippable opt 318.67 -> 302.58
5% raptor-tp6-google-firefox-cold fcp windows10-64-shippable-qr opt 309.54 -> 294.75
5% raptor-tp6-google-firefox-cold windows10-64-shippable-qr opt 406.52 -> 387.32
5% raptor-tp6-google-firefox-cold linux64-shippable-qr opt 424.10 -> 404.23
5% raptor-tp6-bing-firefox-cold loadtime windows10-64-shippable-qr opt 318.83 -> 304.17
5% raptor-tp6-office-firefox-cold fcp linux64-shippable-qr opt 485.58 -> 463.50
5% raptor-tp6-tumblr-firefox loadtime linux64-shippable opt 669.54 -> 639.08
5% raptor-tp6-bing-firefox-cold loadtime linux64-shippable-qr opt 333.58 -> 318.58
4% raptor-tp6-microsoft-firefox-cold linux64-shippable opt 556.21 -> 531.72
4% raptor-tp6-paypal-firefox-cold loadtime linux64-shippable opt 748.92 -> 716.33
4% raptor-tp6-yahoo-mail-firefox-cold fcp windows7-32-shippable opt 304.83 -> 291.67
4% raptor-tp6-microsoft-firefox-cold fcp windows10-64-shippable opt 468.38 -> 448.75
4% raptor-tp6-bing-firefox-cold linux64-shippable-qr opt 258.41 -> 247.62
4% raptor-tp6-imgur-firefox-cold linux64-shippable opt 877.78 -> 841.08
4% raptor-tp6-tumblr-firefox loadtime linux64-shippable-qr opt 696.83 -> 667.79
4% raptor-tp6-office-firefox-cold linux64-shippable-qr opt 491.40 -> 471.64
4% raptor-tp6-yahoo-mail-firefox-cold fcp linux64-shippable-qr opt 368.71 -> 354.08
4% raptor-tp6-facebook-firefox-cold fcp linux64-shippable opt 669.71 -> 643.83
4% raptor-tp6-bing-firefox-cold fcp linux64-shippable-qr opt 256.92 -> 247.25
4% raptor-tp6-youtube-firefox-cold linux64-shippable opt 823.23 -> 792.52
4% raptor-tp6-netflix-firefox-cold fcp linux64-shippable opt 593.42 -> 572.25
4% raptor-tp6-microsoft-firefox-cold fcp windows7-32-shippable opt 469.25 -> 452.58
4% raptor-tp6-office-firefox-cold loadtime linux64-shippable-qr opt 572.83 -> 552.58
4% raptor-tp6-reddit-firefox-cold linux64-shippable-qr opt 760.15 -> 733.22
4% raptor-tp6-reddit-firefox-cold windows7-32-shippable opt 752.83 -> 726.20
3% raptor-tp6-linkedin-firefox windows10-64-shippable-qr opt 739.87 -> 714.39
3% raptor-tp6-twitter-firefox-cold fcp windows10-64-shippable opt 580.00 -> 560.67
3% raptor-tp6-outlook-firefox-cold loadtime windows10-64-shippable opt 625.08 -> 604.67
3% raptor-tp6-netflix-firefox-cold fcp windows7-32-shippable opt 565.33 -> 546.92
3% raptor-tp6-yahoo-mail-firefox-cold linux64-shippable-qr opt 549.00 -> 531.10
3% raptor-tp6-facebook-firefox-cold linux64-shippable opt 856.18 -> 828.47
3% raptor-tp6-reddit-firefox-cold windows10-64-shippable-qr opt 732.16 -> 708.90
3% raptor-tp6-microsoft-firefox-cold fcp linux64-shippable-qr opt 564.75 -> 547.17
3% raptor-tp6-twitter-firefox-cold linux64-shippable opt 748.22 -> 725.14
3% raptor-tp6-twitter-firefox-cold windows10-64-shippable opt 701.97 -> 680.42
3% raptor-tp6-microsoft-firefox-cold linux64-shippable-qr opt 569.38 -> 551.97
3% raptor-tp6-twitter-firefox-cold fcp windows7-32-shippable opt 579.67 -> 561.92
3% raptor-tp6-netflix-firefox-cold linux64-shippable opt 770.97 -> 747.48
3% raptor-tp6-twitter-firefox-cold fcp linux64-shippable-qr opt 608.62 -> 590.17
3% raptor-tp6-netflix-firefox-cold windows7-32-shippable opt 728.50 -> 706.78
3% raptor-tp6-twitter-firefox-cold windows7-32-shippable opt 698.98 -> 678.17
3% raptor-tp6-outlook-firefox-cold loadtime windows7-32-shippable opt 621.25 -> 603.00
3% raptor-tp6-linkedin-firefox windows10-64-shippable opt 729.95 -> 708.78
3% raptor-tp6-facebook-firefox-cold fcp windows7-32-shippable opt 645.25 -> 627.92
3% raptor-tp6-twitter-firefox fcp windows10-64-shippable-qr opt 297.88 -> 289.88
3% raptor-tp6-google-firefox fcp linux64-shippable opt 236.33 -> 230.17
3% raptor-tp6-netflix-firefox-cold fcp linux64-shippable-qr opt 627.25 -> 610.92
3% raptor-tp6-netflix-firefox-cold windows10-64-shippable-qr opt 738.11 -> 719.52
2% raptor-tp6-outlook-firefox-cold loadtime windows10-64-shippable-qr opt 619.67 -> 604.33
2% raptor-tp6-sheets-firefox-cold fcp windows7-32-shippable opt 786.92 -> 767.58
2% raptor-tp6-youtube-firefox-cold loadtime windows10-64-shippable-qr opt 1,466.92 -> 1,431.83
2% raptor-tp6-facebook-firefox-cold windows7-32-shippable opt 804.39 -> 785.69
2% raptor-tp6-sheets-firefox-cold windows7-32-shippable opt 909.65 -> 889.32
2% raptor-tp6-slides-firefox-cold linux64-shippable opt 1,215.88 -> 1,188.70
2% raptor-tp6-twitter-firefox windows10-64-shippable-qr opt 366.88 -> 358.66
2% raptor-tp6-docs-firefox-cold fcp linux64-shippable opt 1,051.96 -> 1,028.83
2% raptor-tp6-yahoo-mail-firefox-cold loadtime linux64-shippable opt 944.75 -> 924.42
2% raptor-tp6-docs-firefox-cold linux64-shippable opt 1,147.70 -> 1,124.47
2% raptor-tp6-slides-firefox-cold fcp linux64-shippable opt 1,192.62 -> 1,168.58

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23556

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

There's also a talos bug open for this regression (1590763) but they are not duplicates as, from a shreiffing point of view, we can't group talos and raptor regressions in a single bug.

Comment 2

[Jim Mathies [:jimm]]

[Tracking Requested - why for this release]:
Serious performance regression in 72.

Hey Matt, is DocumentChannel shipping?

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

From talking with Neha, sounds like DocumentChannel (bug 1583700) is nightly only .

Comment 4

[Matt Woodrow (:mattwoodrow)]

I believe this has been fixed by bug 1590763, but there's some confusion in that bug that still needs to be resolved.

Comment 5

[Dragana Damjanovic [:dragana]]

Matt, I am giving you the bug to you. (I just want such bugs to have an owner.)

Comment 6

[Julien Cristau [:jcristau] (back April 22)]

nightly-only, not tracking.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months and this bug has priority 'P2'.
:dragana, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 8

[Dragana Damjanovic [:dragana]]

Closing the bug as it is fixed before the feature has been shipped.