1523292 - 13.73% raptor-tp6-microsoft-firefox (osx-10-10) regression on push eec1a6963a9cc61b02611259233f54fd15094792 (Sat Jan 26 2019)

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=eec1a6963a9cc61b02611259233f54fd15094792

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

14% raptor-tp6-microsoft-firefox osx-10-10 opt 2,317.37 -> 2,635.43

Improvements:

63% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime osx-10-10 opt 692.54 -> 253.67
54% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime linux64 opt 338.83 -> 154.29
54% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime windows10-64-qr opt 800.00 -> 364.67
54% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime linux64-qr opt 332.75 -> 152.75
54% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime windows10-64 pgo 269.79 -> 124.71
54% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime windows10-64 opt 810.17 -> 376.38
53% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime windows7-32 pgo 705.38 -> 330.25
53% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime windows10-64 pgo 735.79 -> 344.58
53% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime windows10-64 opt 290.25 -> 136.17
53% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime linux64 pgo 295.46 -> 139.21
53% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime windows10-64-qr opt 286.42 -> 135.42
52% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime windows7-32 pgo 261.38 -> 126.00
52% raptor-tp6-google-firefox raptor-tp6-google-firefox-loadtime windows7-32 opt 284.62 -> 137.42
51% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime windows7-32 opt 769.83 -> 378.92
50% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime osx-10-10 opt 1,802.71 -> 899.92
42% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime linux64-qr opt 881.12 -> 513.08
40% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime linux64 opt 832.46 -> 495.92
40% raptor-tp6-wikia-firefox raptor-tp6-wikia-firefox-loadtime osx-10-10 opt 710.33 -> 425.79
40% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-loadtime linux64 pgo 753.33 -> 455.42
39% raptor-tp6-wikia-firefox raptor-tp6-wikia-firefox-loadtime linux64 pgo 292.50 -> 177.46
38% raptor-tp6-wikia-firefox raptor-tp6-wikia-firefox-loadtime linux64 opt 315.25 -> 195.04
38% raptor-tp6-wikia-firefox raptor-tp6-wikia-firefox-loadtime windows7-32 pgo 289.79 -> 179.67
31% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime osx-10-10 opt 757.92 -> 519.38
22% raptor-tp6-amazon-firefox osx-10-10 opt 1,158.05 -> 903.64
20% raptor-tp6-google-firefox osx-10-10 opt 355.98 -> 284.55
18% raptor-tp6-google-firefox linux64 opt 175.74 -> 143.42
18% raptor-tp6-google-firefox windows10-64 opt 156.12 -> 128.35
18% raptor-tp6-amazon-firefox windows10-64-qr opt 477.12 -> 392.90
17% raptor-tp6-google-firefox windows10-64 pgo 143.83 -> 118.70
17% raptor-tp6-google-firefox windows7-32 pgo 144.46 -> 119.64
17% raptor-tp6-google-firefox windows10-64-qr opt 156.39 -> 129.52
17% raptor-tp6-google-firefox linux64-qr opt 175.04 -> 144.99
17% raptor-tp6-google-firefox linux64 pgo 157.54 -> 130.55
16% raptor-tp6-google-firefox windows7-32 opt 155.58 -> 130.66
16% raptor-tp6-amazon-firefox windows10-64 opt 477.63 -> 401.83
15% raptor-tp6-amazon-firefox windows10-64 pgo 426.70 -> 361.73
14% raptor-tp6-amazon-firefox windows7-32 pgo 405.44 -> 349.11
12% raptor-tp6-amazon-firefox windows7-32 opt 449.31 -> 397.11
11% raptor-tp6-amazon-firefox linux64-qr opt 524.28 -> 466.44
10% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime windows7-32 opt 319.10 -> 285.96
9% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime linux64 opt 323.88 -> 293.33
9% raptor-tp6-amazon-firefox linux64 pgo 431.02 -> 390.96
9% raptor-tp6-amazon-firefox linux64 opt 489.04 -> 444.38
9% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime windows10-64 pgo 305.58 -> 278.12
9% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime windows10-64 opt 324.00 -> 294.92
9% raptor-tp6-yandex-firefox raptor-tp6-yandex-firefox-loadtime windows7-32 pgo 296.60 -> 270.00
8% raptor-tp6-wikia-firefox osx-10-10 opt 403.71 -> 370.67
8% raptor-tp6-wikia-firefox linux64 opt 196.31 -> 181.12
6% raptor-tp6-docs-firefox linux64 pgo 1,189.70 -> 1,122.99
5% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-fcp linux64-qr opt 295.31 -> 281.46
4% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-fcp linux64 opt 277.67 -> 266.08
4% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-fcp windows10-64 pgo 234.10 -> 225.46
4% raptor-tp6-facebook-firefox raptor-tp6-facebook-firefox-loadtime linux64 opt 661.79 -> 637.67
3% raptor-tp6-amazon-firefox raptor-tp6-amazon-firefox-fcp windows10-64-qr opt 265.21 -> 257.58

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18959

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Florin Strugariu [:Bebe]]

:jesup thsi caused a big improvement and a regresion on:

14% raptor-tp6-microsoft-firefox osx-10-10 opt 2,317.37 -> 2,635.43

can you take a look over the 2 bugs and point out which one has generated the alert?

Thanks,
Bebe

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

It's bug 1270059 - the other patch is just mechanisms that this one uses

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Note that this particular test doesn't check loadtime, just dcf/fcp/fnbpaint/ttfi. ttfi regressed ~35%, and dcf regressed around 100% (apparently). I'll look at why.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Note that all other platforms on the microsoft-firefox test (in my try runs at least) showed 9-15% improvement in the geomeaned score. Most platforms show some increase in DCF (though generally small; win64 is 6%)

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

So looking at profiles with and without my change on mac: http://bit.ly/2UoVzzo and http://bit.ly/2UoDL7o I don't see any deferred settimeouts (they would be visible as 'settimeout' or 'settimeout release' in the left column of the marker chart. My patch shouldn't have much if any impact on execution if there aren't settimeouts. So, investigating how this could have an impact (other than random measurement changes due to <whatever> - pgo, link order, page alignment, etc)

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Randell Jesup [:jesup] from comment #5)

So looking at profiles with and without my change on mac: http://bit.ly/2UoVzzo and http://bit.ly/2UoDL7o I don't see any deferred settimeouts (they would be visible as 'settimeout' or 'settimeout release' in the left column of the marker chart. My patch shouldn't have much if any impact on execution if there aren't settimeouts. So, investigating how this could have an impact (other than random measurement changes due to <whatever> - pgo, link order, page alignment, etc)

There are timeouts here (was looking at the wrong profile? or not scrolling?) This site is a mess, however; I've seen (on my mac) 10s loads and 40s loads). The timeouts don't seem to have any appreciable impact overall, and there's no obvious reason they'd affect DCF here - and it's unclear if DCF even has any significant utility here, but since there's no loadtime it has a disproportionate impact on the geomean'd result. It hits a price-tracking site a zillion times. I'm looking at it, but it mostly just looks like a poorly-coded site that's very timing-sensitive in odd ways.

I'm going to suggest we find a better url to measure in automation; preferably one of more import and where we can measure loadtime

Comment 7

[Ionuț Goldan [:igoldan]]

(In reply to Randell Jesup [:jesup] from comment #6)

I'm going to suggest we find a better url to measure in automation; preferably one of more import and where we can measure loadtime

ni? Dave on this.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I suggest we accept this regression. I've opened bug 1525534 for changing the URL, but the values we get will not be a fair comparison to the historic data.