Closed Bug 1681030 Opened 4 years ago Closed 4 years ago

5.57 - 126.17% raptor-tp6-netflix-firefox-cold / raptor-tp6-twitch-firefox-cold / raptor-tp6-yandex-firefox-cold (windows10-64-shippable, windows10-64-shippable-qr) regression on push e4c8bc7feaa60470c026236a7d89881a74b29d34 (Wed December 2 2020)

Categories

(Core :: Layout, defect, P2)

Product:
Core
Component:
Layout
Version:
Firefox 85
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
86 Branch
Tracking Flags:
Tracking Status
firefox-esr78 --- unaffected
firefox83 --- unaffected
firefox84 --- unaffected
firefox85 + fixed
firefox86 + fixed

People

(Reporter: Bebe, Assigned: rmader)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression, Whiteboard: [perf:alert:1])

Attachments

(5 files, 1 obsolete file)

old behaviour.webm
1.88 MB, video/webm
Details
new behaviour.webm
887.13 KB, video/webm
Details
browsertime-outlook-old.webm
1.69 MB, video/webm
Details
browsertime-outlock-new.webm
1.65 MB, video/webm
Details
Bug 1681030 - Fix some regressions from bug 1645528, r=mattwoodrow,emilio
48 bytes, text/x-phabricator-request
Details | Review

Perfherder has detected a raptor performance regression from push e4c8bc7feaa60470c026236a7d89881a74b29d34. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
126% raptor-tp6-twitch-firefox-cold loadtime windows10-64-shippable 1,208.00 -> 2,732.08
124% raptor-tp6-twitch-firefox-cold loadtime windows10-64-shippable 1,133.67 -> 2,542.25
29% raptor-tp6-twitch-firefox-cold windows10-64-shippable 276.56 -> 355.55
9% raptor-tp6-yandex-firefox-cold loadtime windows10-64-shippable-qr webrender 1,615.42 -> 1,763.58
6% raptor-tp6-netflix-firefox-cold loadtime windows10-64-shippable 866.08 -> 914.33

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
8% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable-qr webrender 1,020.00 -> 936.92
8% raptor-tp6-google-firefox-cold loadtime windows10-64-shippable-qr webrender 1,102.00 -> 1,013.08
8% raptor-tp6-google-firefox-cold loadtime windows10-64-shippable 1,130.42 -> 1,039.25
6% raptor-tp6-apple-firefox-cold loadtime windows10-64-shippable 1,040.21 -> 974.17
5% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable 1,763.17 -> 1,676.58
5% raptor-tp6-twitter-firefox-cold loadtime windows10-64-shippable-qr webrender 1,762.46 -> 1,682.08
4% raptor-tp6-office-firefox-cold loadtime windows10-64-shippable 2,288.67 -> 2,193.42

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Component: Performance → Widget: Gtk
Flags: needinfo?(robert.mader)
Product: Testing → Core

Hej, I do have a theory what's going on with the regressions (have been looking at bug 1680743), just figuring out how to reproduce it best locally. Will come up with a try push today.

Flags: needinfo?(robert.mader)

Here's a try push for the raptor-tp6-twitch tests: https://treeherder.mozilla.org/jobs?repo=try&revision=b1e538316828fcc40aa362ee60493ed0a1112b54

Edit: previous build had a typo, new try push: https://treeherder.mozilla.org/jobs?repo=try&revision=1aae799a53762b7ebae9c1bc85b26a2f216eead6

Set release status flags based on info from the regressing bug 1645528

status-firefox83: --- → unaffected
status-firefox84: --- → unaffected
status-firefox-esr78: --- → unaffected

This reinstates the behaviour from before D93173. It arguably vialotas
assumptions in other parts parts of the code, for example sActiveVsyncTimers
may have a wrong value, but we may have parts of the code that rely
on this behaviour (in contrast to properly calling StartTimer()).

Assignee: nobody → robert.mader

Also alerts from browsertime:
== Change summary for alert #28004 (as of Mon, 07 Dec 2020 13:12:25 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
381% outlook loadtime linux64-shippable cold nocondprof 1,270.29 -> 6,106.17
375% outlook loadtime linux64-shippable-qr cold nocondprof webrender 1,279.04 -> 6,074.17
132% twitch loadtime linux64-shippable-qr cold nocondprof webrender 1,026.50 -> 2,377.54
130% twitch loadtime linux64-shippable cold nocondprof 1,022.86 -> 2,355.08
87% google-search loadtime linux64-shippable-qr nocondprof warm webrender 271.17 -> 507.46
87% google-search loadtime linux64-shippable nocondprof warm 270.79 -> 506.00
80% yahoo-news loadtime macosx1014-64-shippable-qr cold nocondprof webrender 3,353.40 -> 6,047.25
69% google-search loadtime macosx1014-64-shippable-qr nocondprof warm webrender 467.35 -> 788.75
64% office loadtime linux64-shippable nocondprof warm 730.08 -> 1,199.83
51% office loadtime linux64-shippable-qr nocondprof warm webrender 797.42 -> 1,202.04
46% outlook linux64-shippable cold nocondprof 293.28 -> 428.86
45% outlook linux64-shippable-qr cold nocondprof webrender 293.87 -> 427.49
42% reddit loadtime android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 1,704.98 -> 2,422.62
32% youtube loadtime linux64-shippable-qr nocondprof warm webrender 897.88 -> 1,186.29
31% youtube loadtime linux64-shippable nocondprof warm 902.92 -> 1,185.79
31% youtube loadtime linux64-shippable-qr nocondprof warm webrender 890.60 -> 1,167.88
28% twitch linux64-shippable cold nocondprof 244.04 -> 312.86
28% cnn loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,623.69 -> 5,904.42
27% instagram loadtime linux64-shippable-qr cold nocondprof webrender 1,118.19 -> 1,419.08
25% amazon loadtime macosx1014-64-shippable-qr cold nocondprof webrender 1,664.06 -> 2,075.88
24% twitch linux64-shippable-qr cold nocondprof webrender 251.66 -> 312.48
24% instagram loadtime linux64-shippable cold nocondprof 1,155.19 -> 1,429.54
24% twitter loadtime macosx1014-64-shippable-qr cold nocondprof webrender 2,982.02 -> 3,688.29
17% yandex loadtime macosx1014-64-shippable-qr cold nocondprof webrender 2,163.56 -> 2,528.83
16% yahoo-news macosx1014-64-shippable-qr cold nocondprof webrender 1,701.01 -> 1,980.22
16% bbc SpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 567.25 -> 658.00
16% bbc PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 972.54 -> 1,127.42
16% reddit loadtime linux64-shippable nocondprof warm 1,693.88 -> 1,961.67
15% outlook loadtime macosx1014-64-shippable-qr nocondprof warm webrender 490.52 -> 564.21
15% google-search linux64-shippable-qr nocondprof warm webrender 235.02 -> 269.41
14% google-search macosx1014-64-shippable-qr nocondprof warm webrender 383.10 -> 438.53
14% outlook ContentfulSpeedIndex linux64-shippable nocondprof warm 769.54 -> 878.08
13% cnn-ampstories loadtime android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,331.19 -> 1,510.42
13% office linux64-shippable nocondprof warm 533.53 -> 605.11
13% youtube linux64-shippable-qr nocondprof warm webrender 422.60 -> 478.99
13% google-search linux64-shippable nocondprof warm 240.07 -> 270.76
12% google-slides macosx1014-64-shippable-qr nocondprof warm webrender 628.07 -> 703.59
12% reddit loadtime linux64-shippable-qr nocondprof warm webrender 1,756.92 -> 1,959.88
11% twitch FirstVisualChange linux64-shippable-qr cold nocondprof webrender 815.00 -> 906.67
10% office linux64-shippable-qr nocondprof warm webrender 546.55 -> 603.55
10% twitter linux64-shippable nocondprof warm 258.89 -> 284.99
9% twitch FirstVisualChange linux64-shippable cold nocondprof 820.00 -> 896.67
9% twitter linux64-shippable-qr nocondprof warm webrender 261.91 -> 285.02
8% facebook FirstVisualChange linux64-shippable cold nocondprof 1,210.00 -> 1,310.00
8% google-slides linux64-shippable cold nocondprof 918.79 -> 991.60
8% reddit LastVisualChange linux64-shippable nocondprof warm 1,941.67 -> 2,093.33
7% cnn-ampstories loadtime android-hw-g5-7-0-arm7-api-16-shippable live nocondprof warm webrender 534.42 -> 571.21
7% facebook SpeedIndex linux64-shippable cold nocondprof 1,239.75 -> 1,325.08
7% google-slides linux64-shippable-qr cold nocondprof webrender 924.88 -> 988.44
7% youtube loadtime android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 914.83 -> 976.25
6% facebook FirstVisualChange linux64-shippable-qr cold nocondprof webrender 1,225.00 -> 1,300.00
6% cnn android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 2,793.04 -> 2,963.73
6% twitter macosx1014-64-shippable-qr cold nocondprof webrender 697.32 -> 739.60
6% facebook ContentfulSpeedIndex linux64-shippable cold nocondprof 1,275.88 -> 1,352.67
5% twitch SpeedIndex linux64-shippable-qr cold nocondprof webrender 1,049.42 -> 1,106.33
5% facebook PerceptualSpeedIndex linux64-shippable cold nocondprof 1,257.83 -> 1,324.00
5% facebook SpeedIndex macosx1014-64-shippable-qr nocondprof warm webrender 618.12 -> 649.75
5% instagram linux64-shippable-qr cold nocondprof webrender 361.69 -> 380.01
5% facebook PerceptualSpeedIndex macosx1014-64-shippable-qr nocondprof warm webrender 644.88 -> 676.58
5% twitch SpeedIndex linux64-shippable cold nocondprof 1,051.42 -> 1,102.42
4% facebook PerceptualSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,259.04 -> 1,314.58
4% twitch PerceptualSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,414.67 -> 1,466.83
4% microsoft loadtime macosx1014-64-shippable-qr nocondprof warm webrender 557.90 -> 577.83
3% twitch PerceptualSpeedIndex linux64-shippable cold nocondprof 1,418.29 -> 1,462.75

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
16% microsoft LastVisualChange macosx1014-64-shippable-qr nocondprof warm webrender 750.00 -> 626.67
14% microsoft LastVisualChange macosx1014-64-shippable-qr cold nocondprof webrender 1,313.33 -> 1,126.67
14% cnn LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 5,719.42 -> 4,941.17
14% instagram PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 620.08 -> 535.67
13% microsoft LastVisualChange linux64-shippable-qr cold nocondprof webrender 863.33 -> 750.00
12% instagram loadtime macosx1014-64-shippable-qr nocondprof warm webrender 939.29 -> 823.42
12% cnn ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,350.71 -> 3,816.25
12% imdb LastVisualChange linux64-shippable-qr nocondprof warm webrender 935.00 -> 823.33
11% cnn SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,970.62 -> 4,412.00
11% cnn PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,529.50 -> 4,020.83
11% imdb LastVisualChange macosx1014-64-shippable-qr nocondprof warm webrender 1,471.67 -> 1,306.67
11% microsoft LastVisualChange linux64-shippable cold nocondprof 851.67 -> 756.67
11% wikipedia LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 497.38 -> 442.00
11% instagram loadtime linux64-shippable-qr nocondprof warm webrender 575.58 -> 513.38
11% instagram loadtime linux64-shippable nocondprof warm 573.19 -> 512.08
11% imdb loadtime linux64-shippable nocondprof warm 881.21 -> 787.71
10% outlook ContentfulSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,929.17 -> 1,727.75
10% google-sheets PerceptualSpeedIndex linux64-shippable nocondprof warm 624.17 -> 561.08
10% google-sheets PerceptualSpeedIndex linux64-shippable-qr nocondprof warm webrender 625.79 -> 562.75
10% google fcp android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 178.21 -> 161.00
9% instagram LastVisualChange android-hw-p2-8-0-android-aarch64-shippable cold nocondprof webrender 1,957.08 -> 1,775.08
9% instagram LastVisualChange android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 1,620.67 -> 1,470.83
9% bbc LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 5,396.21 -> 4,900.33
9% imdb loadtime linux64-shippable-qr nocondprof warm webrender 868.25 -> 789.25
9% ebay-kleinanzeigen PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 651.71 -> 592.92
9% bbc PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,751.67 -> 4,327.33
9% youtube loadtime linux64-shippable cold nocondprof 1,807.29 -> 1,652.29
9% youtube linux64-shippable-qr cold nocondprof webrender 846.78 -> 774.42
8% amazon LastVisualChange linux64-shippable-qr nocondprof warm webrender 1,180.00 -> 1,080.00
8% outlook ContentfulSpeedIndex linux64-shippable cold nocondprof 1,909.92 -> 1,752.08
8% office ContentfulSpeedIndex macosx1014-64-shippable-qr cold nocondprof webrender 2,510.04 -> 2,305.25
8% office LastVisualChange macosx1014-64-shippable-qr cold nocondprof webrender 3,836.67 -> 3,526.67
8% bbc SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,205.08 -> 3,869.17
8% youtube loadtime linux64-shippable-qr cold nocondprof webrender 1,788.40 -> 1,650.62
8% cnn-ampstories fcp android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,293.75 -> 1,195.58
7% cnn-ampstories ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,424.08 -> 1,317.58
7% cnn-ampstories FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,424.08 -> 1,317.58
7% cnn-ampstories LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,424.08 -> 1,317.58
7% cnn-ampstories PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,424.08 -> 1,317.58
7% cnn-ampstories SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,424.08 -> 1,317.58
7% twitter ContentfulSpeedIndex linux64-shippable nocondprof warm 773.33 -> 716.92
7% youtube linux64-shippable cold nocondprof 838.23 -> 777.06
7% linkedin ContentfulSpeedIndex linux64-shippable nocondprof warm 993.62 -> 921.17
7% twitter loadtime linux64-shippable cold nocondprof 1,861.46 -> 1,733.25
7% office ContentfulSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,612.00 -> 1,502.42
6% office ContentfulSpeedIndex linux64-shippable cold nocondprof 1,599.92 -> 1,496.25
6% imdb loadtime macosx1014-64-shippable-qr nocondprof warm webrender 1,247.54 -> 1,167.12
6% apple loadtime macosx1014-64-shippable-qr cold nocondprof webrender 1,486.19 -> 1,398.88
6% twitter loadtime linux64-shippable-qr cold nocondprof webrender 1,842.38 -> 1,735.75
6% instagram LastVisualChange linux64-shippable nocondprof warm 873.33 -> 823.33
6% office LastVisualChange linux64-shippable cold nocondprof 2,518.33 -> 2,376.67
6% office LastVisualChange linux64-shippable-qr cold nocondprof webrender 2,525.00 -> 2,383.33
5% amazon LastVisualChange android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 801.83 -> 758.75
5% google-search loadtime linux64-shippable-qr cold nocondprof webrender 1,036.71 -> 982.46
5% cnn-ampstories loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 2,913.98 -> 2,762.50
5% fandom LastVisualChange linux64-shippable cold nocondprof 1,161.67 -> 1,103.33
5% speedometer android-hw-p2-8-0-android-aarch64-shippable geckoview nocondprof webrender 27.65 -> 29.01
5% google-search loadtime linux64-shippable cold nocondprof 1,030.21 -> 981.04
5% office ContentfulSpeedIndex macosx1014-64-shippable-qr nocondprof warm webrender 1,498.38 -> 1,429.42
5% office loadtime linux64-shippable-qr cold nocondprof webrender 2,589.54 -> 2,470.92
4% linkedin PerceptualSpeedIndex linux64-shippable nocondprof warm 1,276.92 -> 1,220.75
4% twitter PerceptualSpeedIndex linux64-shippable cold nocondprof 1,153.71 -> 1,103.17
4% youtube PerceptualSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,277.92 -> 1,222.50
4% twitter PerceptualSpeedIndex macosx1014-64-shippable-qr cold nocondprof webrender 1,770.83 -> 1,694.50
4% fandom LastVisualChange macosx1014-64-shippable-qr cold nocondprof webrender 1,748.33 -> 1,673.33
4% google loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 1,092.02 -> 1,045.12
4% linkedin PerceptualSpeedIndex linux64-shippable-qr nocondprof warm webrender 1,272.04 -> 1,218.17
4% twitter ContentfulSpeedIndex macosx1014-64-shippable-qr cold nocondprof webrender 2,487.88 -> 2,384.50
4% linkedin LastVisualChange linux64-shippable-qr nocondprof warm webrender 2,336.67 -> 2,240.00
4% linkedin LastVisualChange linux64-shippable nocondprof warm 2,331.67 -> 2,236.67
4% instagram linux64-shippable nocondprof warm 225.03 -> 215.99
4% instagram linux64-shippable-qr nocondprof warm webrender 225.12 -> 216.12
4% office loadtime linux64-shippable cold nocondprof 2,583.62 -> 2,481.54
4% twitter LastVisualChange linux64-shippable cold nocondprof 2,723.33 -> 2,616.67
4% twitter PerceptualSpeedIndex linux64-shippable-qr cold nocondprof webrender 1,150.67 -> 1,106.42
4% ebay-kleinanzeigen LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,750.58 -> 3,606.42
4% fandom SpeedIndex linux64-shippable cold nocondprof 863.50 -> 830.75
4% yahoo-news ContentfulSpeedIndex macosx1014-64-shippable-qr nocondprof warm webrender 787.42 -> 757.58
4% twitter LastVisualChange linux64-shippable-qr cold nocondprof webrender 2,716.67 -> 2,620.00
3% twitter ContentfulSpeedIndex linux64-shippable cold nocondprof 1,621.33 -> 1,567.92
3% office PerceptualSpeedIndex macosx1014-64-shippable-qr cold nocondprof webrender 1,781.00 -> 1,726.17
3% google-slides ContentfulSpeedIndex linux64-shippable cold nocondprof 1,214.67 -> 1,179.58
3% twitter SpeedIndex linux64-shippable cold nocondprof 1,692.79 -> 1,644.92
3% instagram PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,251.96 -> 1,218.08
2% instagram LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,525.50 -> 3,446.83
2% google-sheets loadtime linux64-shippable-qr nocondprof warm webrender 986.92 -> 967.17

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28004

Component: Widget: Gtk → Layout
See Also: → 1680743

I was just able to reproduce the biggest regression locally, i.e.

381% 	outlook 	loadtime 	linux64-shippable 	cold nocondprof 	1,270.29 -> 6,106.17

assuming ./mach raptor -t raptor-tp6-outlook-firefox --enable-webrender --cold is the correct command (I forgot --no-conditioned-profile, but I guess it doesn't make a bit difference here).

What strikes me is that the visual behaviour appears to be quite different. Florin, do you know who would be a good person to talk to about the test? I'm currently trying to understand what exactly we are actually measuring, as the new slower behaviour arguably looks more "correct".

Will attach some videos.

Flags: needinfo?(fstrugariu)
Attached video old behaviour.webm —
Attached video new behaviour.webm —
Attachment #9191629 - Attachment is obsolete: true

FTR., here are the numbers I get for the outlook test

current tip:
[5336, 5362, 5439, 5487, 5300, 7960, 1590, 1599, 7951, 5473, 5622, 5382, 5471, 8011, 1676, 5819, 5252, 5659, 5418, 5111, 5220, 5416, 1685, 5444, 5237]
pre 1645528:
[1722, 1723, 1664, 1610, 1570, 1640, 2595, 1575, 1643, 1709, 1749, 1621, 1610, 1781, 1638, 1670, 1588, 1752, 1788, 1568, 1617, 1639, 1710, 1661, 1610]

What kinda jumps to attention is that the test sometimes still finishes around the 1600 mark, as it did previously. But usually it will take a while longer and load more things. I wonder if this is another case of flakiness, but with a strong tendency in one direction. And if the new results are maybe the "real" or desired ones.

This is (in "current") a bimodal distribution. This isn't all that unusual; for example if a timer fires, and it's before onload, and it loads some large resource, that will block onload. If it fires after onload, it obviously doesn't. If the timer fires somewhere near onload, it could fall on either side due to noise.

This overall set of regressions is quite large....

:Sparky maybe you can help with some context on when these metric should fire exactly

also here are some information on the mtrics we gather:
https://wiki.mozilla.org/Performance/Glossary#Pageload
https://www.sitespeed.io/documentation/sitespeed.io/metrics/#pageloadtime

Flags: needinfo?(fstrugariu) → needinfo?(gmierz2)
Whiteboard: [perf:alert:?]

(In reply to Robert Mader [:rmader] from comment #6)

I was just able to reproduce the biggest regression locally, i.e.

381% 	outlook 	loadtime 	linux64-shippable 	cold nocondprof 	1,270.29 -> 6,106.17

assuming ./mach raptor -t raptor-tp6-outlook-firefox --enable-webrender --cold is the correct command (I forgot --no-conditioned-profile, but I guess it doesn't make a bit difference here).

What strikes me is that the visual behaviour appears to be quite different. Florin, do you know who would be a good person to talk to about the test? I'm currently trying to understand what exactly we are actually measuring, as the new slower behaviour arguably looks more "correct".

Will attach some videos.

you are running raptor webextension tests not browsertime (platform on which we logged the regression)
to run the test in browsertime you just need to add --brosertime in the command line

so the command is:
./mach raptor -t raptor-tp6-outlook-firefox --enable-webrender --cold --browsertime

Flags: needinfo?(robert.mader)

Thanks, that was an important hint. With --browsertime the case is a little bit different: now both cases seem to visually behave pretty much equally - however, the reported loadtime still heavily diverges (roughly 1.6 vs 6 seconds, see videos).

I also checked some of the other heavily impacted test, especially twitch, and IIUC they all load elements via JS.
To me, I'm increasingly tending to the following conclusion:

  1. the major regression appear to be all for loadtime, on sites where loadtime may not be a meaningful metric. They all rely on AJAX, which probably means the sentence "Important: this is only relevant to some pages, depending on how you page is built." from https://www.sitespeed.io/documentation/sitespeed.io/metrics/#pageloadtime applies to them.
  2. the rest of regressions/improvements appears to equal each other out, more or less

Does that sound valid? :/

Flags: needinfo?(robert.mader)

Olli, does the above sound reasonable to you?

Flags: needinfo?(bugs)

That sounds reasonable, but it doesn't mean we should take major page load regressions.
Page load is after all the time when browser UI shows that the page has been loaded.

At least we should understand what is causing these changes in page load.
Given that there is a merge next week, should the regressing patch(es) be backed out for now?

Flags: needinfo?(bugs)

Comment on attachment 9192261 [details]
browsertime-outlook-old.webm

The NEW behaviour (sorry wrong file name)

Severity: -- → S2
Priority: -- → P2

For the record: the previous behaviour can be restored by using a software timer instead of the vsyncchild. I'm currently trying to figure out what exactly behaves different between the two, with some theories I'm trying to verify.

It'd be good to know this week if we should back out bug 1645528 for 85.

status-firefox86: --- → affected
tracking-firefox85: --- → +

(In reply to Julien Cristau [:jcristau] from comment #20)

It'd be good to know this week if we should back out bug 1645528 for 85.

Yes, please back out, at least for beta. I'm still investigating the issue so you could consider leaving it in for nightly - if that's possible (see also bug 1645528 comment 68).

Regressing patch backed out of 85:
https://hg.mozilla.org/releases/mozilla-beta/rev/01efe9d5b4e1006d7908576c8540139b633294df

status-firefox85: affected → fixed
tracking-firefox86: --- → +
Flags: needinfo?(gmierz2)
Whiteboard: [perf:alert:?] → [perf:alert:1]

This fixes a bunch of regressions:

  • a wrong calculation in GetIdleDeadlineHint(), leading to pageload
    regressions.
  • in certain situations we'd use StartupRefreshDriverTimer instead
    of VsyncRefreshDriverTimer when initializing timers early
  • unnecessary use of BrowserChild on backends that don't opt for
    per-browser-child vsync - i.e. all but Wayland.

This is partly done by reverting to pre-1645528 behaviour, although
with some code simplifications.

FTR: I also played with some more radical changes, but given the
complexity of the code involved I found the regression potential too
big. Thus this is the most conservative solution I could come up with.

Try job: https://treeherder.mozilla.org/jobs?repo=try&revision=04be8d989a48dbc8342cd84ffc39e56629be18e8

Also checks for the regressions in bug 1680743 and bug 1673139

The try run looks good from what I can see - e.g. outlook loadtime, the biggest regression here, is back to 1287.5ms (https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=try,2912794,1,13&timerange=1209600), twitch at 1035.5ms.

Pushed by cbrindusan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f64bb4f21c91
Fix some regressions from bug 1645528, r=emilio

https://hg.mozilla.org/mozilla-central/rev/f64bb4f21c91

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox86: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
Regressions: 1684352

Backed out for causing crashes for Linux users:

https://hg.mozilla.org/integration/autoland/rev/f237c5e2a071dcaae542a34afc3c463099d5b1df

Crash example: bp-6d7050f8-c862-4a35-b396-179470201228

Status: RESOLVED → REOPENED
Crash Signature: [@ __pthread_mutex_lock | mozilla::detail::MutexImpl::lock | mozilla::RefreshTimerVsyncDispatcher::AddChildRefreshTimer]
status-firefox86: fixed → ---
Flags: needinfo?(robert.mader)
Resolution: FIXED → ---
Target Milestone: 86 Branch → ---
Pushed by rmaries@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/59a0ddad5c28
Fix some regressions from bug 1645528, r=emilio

Err yes, a fix for that crashes is already landing in bug 1684352 - so, right, you can simply re-land, as just happened :)

Flags: needinfo?(robert.mader)

https://hg.mozilla.org/mozilla-central/rev/59a0ddad5c28

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox86: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
Crash Signature: [@ __pthread_mutex_lock | mozilla::detail::MutexImpl::lock | mozilla::RefreshTimerVsyncDispatcher::AddChildRefreshTimer]

== Change summary for alert #28291 (as of Wed, 30 Dec 2020 00:38:48 GMT) ==

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
57% raptor-tp6-twitch-firefox-cold loadtime windows10-64-shippable 2,661.46 -> 1,135.67
23% raptor-tp6-twitch-firefox-cold windows10-64-shippable 350.93 -> 270.16

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28291

== Change summary for alert #28307 (as of Thu, 31 Dec 2020 08:38:26 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
7% tp5o_webext windows10-64-shippable-qr e10s stylo webrender-sw 335.43 -> 359.42
5% tp5o_webext linux64-shippable-qr e10s stylo webrender 285.63 -> 298.50
4% tp5o_webext linux64-shippable e10s stylo 275.11 -> 286.12
4% tp5o_webext linux64-shippable-qr e10s stylo webrender 287.24 -> 297.68

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
28% about_preferences_basic linux64-shippable e10s stylo 167.11 -> 119.69
28% about_preferences_basic windows10-64-shippable-qr e10s stylo webrender-sw 170.98 -> 123.16
20% about_preferences_basic linux64-shippable-qr e10s stylo webrender-sw 154.29 -> 123.00
19% about_preferences_basic linux64-shippable-qr e10s stylo webrender 159.08 -> 128.92
14% tp5o responsiveness linux64-shippable e10s stylo 0.95 -> 0.82
14% tp5o responsiveness linux64-shippable-qr e10s stylo webrender-sw 1.05 -> 0.91
9% tp5o_webext responsiveness linux64-shippable e10s stylo 1.61 -> 1.46

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28307

== Change summary for alert #28312 (as of Thu, 31 Dec 2020 13:03:19 GMT) ==

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
3% JS linux1804-64-shippable-qr tp6 174,098,924.41 -> 169,447,251.04
1% JS linux1804-64-shippable-qr tp6 173,967,514.33 -> 171,884,995.60

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28312

== Change summary for alert #28292 (as of Wed, 30 Dec 2020 00:54:24 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
18% cnn LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,964.33 -> 5,854.83
17% cnn ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,809.92 -> 4,452.33
16% cnn SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,397.42 -> 5,091.25
15% cnn PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 4,013.42 -> 4,633.83
10% web-de ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 375.88 -> 411.75
9% google-search LastVisualChange linux64-shippable cold nocondprof 366.67 -> 400.00
8% apple loadtime macosx1014-64-shippable-qr cold nocondprof webrender 1,405.83 -> 1,522.25
8% google-slides LastVisualChange linux64-shippable-qr cold nocondprof webrender 4,333.33 -> 4,670.00
8% cnn-ampstories ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,291.42 -> 1,390.17
8% cnn-ampstories FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,291.42 -> 1,390.17
8% cnn-ampstories LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,291.42 -> 1,390.17
8% cnn-ampstories PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,291.42 -> 1,390.17
8% cnn-ampstories SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,291.42 -> 1,390.17
8% google-slides loadtime linux64-shippable-qr cold nocondprof webrender 4,331.42 -> 4,657.04
6% cnn-ampstories loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 2,837.32 -> 3,017.96
5% espn android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,291.67 -> 3,447.82
5% instagram LastVisualChange linux64-shippable nocondprof warm 800.00 -> 836.67
4% cnn FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,446.42 -> 3,593.33
4% google-slides ContentfulSpeedIndex linux64-shippable-qr nocondprof warm webrender 601.62 -> 624.33
4% cnn-ampstories LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,139.96 -> 3,257.83
4% cnn-ampstories ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,099.25 -> 3,213.00
4% cnn-ampstories FirstVisualChange android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 3,099.25 -> 3,213.00
3% speedometer android-hw-g5-7-0-arm7-api-16-shippable nocondprof webrender 10.67 -> 10.33
3% instagram linux64-shippable nocondprof warm 212.87 -> 219.03
3% instagram ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 875.29 -> 899.58
3% instagram PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 1,210.08 -> 1,243.25
2% instagram LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 2,390.25 -> 2,443.83
2% instagram SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable nocondprof warm webrender 2,098.83 -> 2,145.83

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
24% cnn loadtime android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 5,839.92 -> 4,429.58
21% bbc loadtime android-hw-p2-8-0-android-aarch64-shippable nocondprof warm webrender 784.81 -> 623.83
6% cnn android-hw-g5-7-0-arm7-api-16-shippable cold nocondprof webrender 2,934.89 -> 2,745.67
5% google-slides macosx1014-64-shippable-qr nocondprof warm webrender 653.88 -> 619.89
2% wikipedia loadtime linux64-shippable nocondprof warm 601.90 -> 588.96

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=28292

Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: