1516962 - big perf changes when landing loadtime measurement

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

== Change summary for alert #18520 (as of Fri, 28 Dec 2018 19:01:08 GMT) ==

Regressions:

 48%  raptor-tp6-reddit-firefox osx-10-10 opt              1,561.60 -> 2,306.17
 32%  raptor-tp6-reddit-firefox windows10-64 pgo           776.17 -> 1,027.12
 31%  raptor-tp6-imdb-firefox osx-10-10 opt                332.81 -> 434.77
 30%  raptor-tp6-reddit-firefox windows7-32 pgo            755.68 -> 983.21
 30%  raptor-tp6-reddit-firefox windows10-64-qr opt        872.53 -> 1,132.97
 29%  raptor-tp6-reddit-firefox windows10-64 opt           897.15 -> 1,155.55
 29%  raptor-tp6-reddit-firefox linux64 pgo                747.68 -> 962.37
 27%  raptor-tp6-reddit-firefox linux64-qr opt             864.99 -> 1,099.80
 27%  raptor-tp6-reddit-firefox windows7-32 opt            859.43 -> 1,090.93
 25%  raptor-tp6-yandex-firefox osx-10-10 opt              306.42 -> 383.08
 23%  raptor-tp6-yandex-firefox windows10-64 opt           158.07 -> 195.12
 23%  raptor-tp6-yandex-firefox windows10-64 pgo           150.10 -> 184.87
 23%  raptor-tp6-youtube-firefox linux64 pgo               329.07 -> 404.37
 23%  raptor-tp6-youtube-firefox windows7-32 pgo           322.28 -> 395.05
 22%  raptor-tp6-youtube-firefox windows10-64 pgo          333.47 -> 407.07
 22%  raptor-tp6-youtube-firefox windows10-64-qr opt       349.86 -> 426.05
 22%  raptor-tp6-yandex-firefox windows7-32 opt            158.81 -> 193.28
 22%  raptor-tp6-youtube-firefox windows7-32 opt           343.35 -> 417.64
 22%  raptor-tp6-youtube-firefox linux64-qr opt            381.36 -> 463.73
 22%  raptor-tp6-imdb-firefox linux64-qr opt               302.16 -> 367.19
 21%  raptor-tp6-yandex-firefox linux64-qr opt             148.26 -> 180.04
 21%  raptor-tp6-youtube-firefox linux64 opt               360.05 -> 436.09
 21%  raptor-tp6-yandex-firefox windows7-32 pgo            149.67 -> 180.98
 21%  raptor-tp6-yandex-firefox linux64 opt                153.79 -> 185.80
 20%  raptor-tp6-yandex-firefox linux64 pgo                144.26 -> 172.86
 20%  raptor-tp6-wikia-firefox osx-10-10 opt               373.41 -> 446.46
 19%  raptor-tp6-yandex-firefox windows10-64-qr opt        142.00 -> 169.48
 19%  raptor-tp6-imdb-firefox linux64 pgo                  272.93 -> 324.13
 18%  raptor-tp6-imdb-firefox linux64 opt                  294.37 -> 347.35
 18%  raptor-tp6-wikia-firefox linux64 pgo                 157.84 -> 186.05
 18%  raptor-tp6-youtube-firefox windows10-64 opt          357.35 -> 421.10
 17%  raptor-tp6-wikia-firefox windows10-64-qr opt         176.10 -> 206.21
 17%  raptor-tp6-google-firefox linux64-qr opt             158.21 -> 185.03
 16%  raptor-tp6-wikia-firefox linux64 opt                 183.76 -> 212.77
 16%  raptor-tp6-google-firefox windows10-64 opt           143.01 -> 165.51
 15%  raptor-tp6-wikia-firefox windows7-32 pgo             157.08 -> 181.35
 15%  raptor-tp6-wikia-firefox linux64-qr opt              193.03 -> 222.81
 15%  raptor-tp6-google-firefox windows10-64-qr opt        142.75 -> 164.33
 15%  raptor-tp6-google-firefox linux64 pgo                145.60 -> 167.52
 15%  raptor-tp6-google-firefox linux64 opt                160.60 -> 184.30
 15%  raptor-tp6-google-firefox windows7-32 pgo            133.14 -> 152.62
 14%  raptor-tp6-google-firefox windows7-32 opt            144.73 -> 165.33
 14%  raptor-tp6-wikia-firefox windows7-32 opt             177.15 -> 201.58
 13%  raptor-tp6-google-firefox windows10-64 pgo           132.94 -> 150.75
 13%  raptor-tp6-wikia-firefox windows10-64 opt            182.72 -> 206.87
 12%  raptor-tp6-facebook-firefox windows7-32 opt          388.74 -> 435.47
 12%  raptor-tp6-facebook-firefox linux64 pgo              365.66 -> 409.39
 12%  raptor-tp6-facebook-firefox windows10-64 pgo         376.55 -> 421.08
 12%  raptor-tp6-amazon-firefox windows7-32 pgo            410.41 -> 458.58
 12%  raptor-tp6-facebook-firefox windows7-32 pgo          369.28 -> 412.58
 12%  raptor-tp6-amazon-firefox linux64 opt                477.90 -> 533.33
 11%  raptor-tp6-facebook-firefox linux64 opt              393.60 -> 437.68
 11%  raptor-tp6-facebook-firefox linux64-qr opt           415.12 -> 461.00
 11%  raptor-tp6-amazon-firefox linux64 pgo                429.43 -> 476.63
 11%  raptor-tp6-facebook-firefox windows10-64-qr opt      409.17 -> 453.83
 11%  raptor-tp6-facebook-firefox windows10-64 opt         398.60 -> 442.04
 11%  raptor-tp6-amazon-firefox windows10-64 opt           478.67 -> 530.51
 11%  raptor-tp6-amazon-firefox windows10-64-qr opt        480.17 -> 531.71
 10%  raptor-tp6-amazon-firefox windows7-32 opt            450.77 -> 497.84
 10%  raptor-tp6-amazon-firefox windows10-64 pgo           429.75 -> 473.90
 10%  raptor-tp6-amazon-firefox linux64-qr opt             525.85 -> 575.94
  9%  raptor-tp6-amazon-firefox osx-10-10 opt              1,228.52 -> 1,342.70

Improvements:

 33%  raptor-tp6-sheets-firefox windows7-32 pgo          711.52 -> 478.68
 31%  raptor-tp6-docs-firefox linux64 opt                1,836.25 -> 1,259.66
 31%  raptor-tp6-docs-firefox linux64-qr opt             1,913.67 -> 1,319.90
 31%  raptor-tp6-sheets-firefox windows10-64 opt         734.27 -> 509.36
 30%  raptor-tp6-sheets-firefox windows10-64-qr opt      756.28 -> 525.97
 30%  raptor-tp6-docs-firefox linux64 pgo                1,703.85 -> 1,196.12
 30%  raptor-tp6-sheets-firefox windows7-32 opt          725.60 -> 510.27
 29%  raptor-tp6-docs-firefox osx-10-10 opt              3,859.92 -> 2,730.34
 29%  raptor-tp6-sheets-firefox linux64 opt              730.49 -> 517.14
 28%  raptor-tp6-sheets-firefox windows10-64 pgo         712.85 -> 511.83
 27%  raptor-tp6-sheets-firefox linux64-qr opt           775.54 -> 568.49
 25%  raptor-tp6-sheets-firefox osx-10-10 opt            1,582.15 -> 1,193.48
 23%  raptor-tp6-sheets-firefox linux64 pgo              673.53 -> 519.89
 22%  raptor-tp6-docs-firefox windows10-64 opt           1,987.18 -> 1,549.37
 21%  raptor-tp6-docs-firefox windows10-64-qr opt        1,975.43 -> 1,558.89
 20%  raptor-tp6-docs-firefox windows7-32 pgo            1,811.20 -> 1,450.18
 20%  raptor-tp6-docs-firefox windows7-32 opt            1,876.90 -> 1,503.58
 19%  raptor-tp6-docs-firefox windows10-64 pgo           1,870.08 -> 1,512.99
 18%  raptor-tp6-slides-firefox windows10-64-qr opt      1,411.89 -> 1,164.00
 17%  raptor-tp6-slides-firefox windows7-32 opt          1,370.99 -> 1,132.57
 17%  raptor-tp6-slides-firefox windows10-64 opt         1,438.36 -> 1,200.07
 15%  raptor-tp6-slides-firefox windows10-64 pgo         1,318.44 -> 1,119.38
 15%  raptor-tp6-slides-firefox linux64 pgo              1,179.26 -> 1,002.08
 15%  raptor-tp6-slides-firefox linux64-qr opt           1,361.92 -> 1,157.82
 14%  raptor-tp6-slides-firefox linux64 opt              1,269.66 -> 1,091.22
 14%  raptor-tp6-slides-firefox windows7-32 pgo          1,258.83 -> 1,084.13
 12%  raptor-tp6-slides-firefox osx-10-10 opt            3,735.99 -> 3,270.90

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=18520

this all comes from bug 1504013:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=295df756fd85d8a66ffd39b637752320e1e8286d&tochange=bc116e143c5619a39423a104538bc62293323186

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:rwood, can you confirm/resolve this?

Comment 2

[Robert Wood [:rwood]]

Thanks :jmaher. Yes these are all as expected, we are now measuring 'loadtime' which takes a long time to get - so it has increased all the geometric mean values.

The improvements are because I turned off ttfi (and not measuring loadtime on the google docs/slides/sheets tests, because it takes > 30 seconds on each pageload to retrieve those, making the job times too long. If dev notes that it is absolutely necessary to measure ttfi and loadtime on the google docs/slides/sheets tests then in the future we could break up that test suite (tp6-2).