2013916 - Significant benchmark regressions in ubuntu 24.04

Status: NEW

(Testing :: Performance, task, P3)

Description

[Kash Shampur [:kshampur] ⌚EST]

for reference:

• Motionmark 1.3: 50-75% regression
• Talos glterrain: 300% regression
• wasm-godot-* : 17-33% regression

Looking at the profiles for e.g. motionmark, parent process memory usage is 3-4x higher on 24.04:
  - Before (18.04): https://share.firefox.dev/4a5ZNo1
  - After (24.04): https://share.firefox.dev/3LEmwj1

Perfcompares:

• motionmark and wasm-godot: https://perf.compare/compare-results?baseRev=5c3571269763df5151d0b7671331add83448c175&baseRepo=try&newRev=784c6ef6e767b252407a661a1f0c46d1f27762bb&newRepo=try&framework=13&test_version=student-t&filter_confidence=high&filter_status=regression
• glterrain https://perf.compare/compare-results?baseRev=63464647b9c76d4c9014dfa4c0a6d4d894009ca8&baseRepo=try&newRev=3b62bd337a59a79ca98a73fb1ed79fb2476512ff&newRepo=try&framework=1&filter_status=regression&sort=delta%7Cdesc&search=glterrain

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

Attachment: before.png

18.04 profile memory

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

Attachment: after.png

24.04 profile memory

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

Attachment: perfcompare.png

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

Attachment: glterrain.png

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

Interestingly mentioned by :nordzilla re: translation tests https://bugzilla.mozilla.org/show_bug.cgi?id=2013917#c2

Basically increase in parent process memory may not be a concern at all? Something to keep in mind

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

profile for gl terrain (webgl) available at these task links

18.04

24.04

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

this came up in other channels so just making note in this bug, the machines still use x11 in ubuntu 24.04
we will need to make some changes upstream to enable wayland + browsertime (Bug 1978662) afterwards

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

the webgl profiles here should have the update symbolicator-cli (so inline views)

• 18.04: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=webgl&revision=4a2850205796521e7d698767cfff9e15426859d7
• 24.04: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=PPObitpcTC6ajM0M-lYsPQ.0&tier=1%2C2%2C3&searchStr=webgl&revision=76c0cb0961a3d129b747dd20499a1ea2caa2e286

Unfortunately for motionmark, I recall there is a bit of an issue with raptor tests that use --extra-profiler-run flag and using the new samply/symbolicator-cli setup, and that is tracked in Bug 2000026.
So for now the motionmark profiles are not using symbolicator-cli

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

Summarizing a convo I had with Markus (might be missing some details but general gist...)

• Difference in OS version might be using different graphics drivers.
• Motionmark profiles don't have the updated symbolication (as stated above) but that would help going forward for future investigation
• Basically nothing in this patch suggests preventing the landing Bug 2002063 as "worse" data is still more useful if it is more up to date (e.g. OS version)