1902979 - svg/animations/end-of-time-*-crash.html wpt tests run out of memory on linux CI

Status: RESOLVED FIXED

(Testing :: CI Configuration, defect)

Description

[Julien Cristau [:jcristau]]

The web-platform-tests-crashtest task on linux CI appears to frequently run out of memory. On the old X11 / ubuntu 18.04 workers, the task survives after a 45s timeout. On the newer wayland / ubuntu 22.04 workers, especially with a debug build, more often than not the worker itself seems to run OOM.

https://profiler.firefox.com/from-url/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FCvcJevkOSbif9Fc73EUfGg%2Fruns%2F2%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_resource-usage.json/marker-chart/?globalTrackOrder=0&thread=0&timelineType=stack&v=10 is from one of the rare runs on ubuntu 22.04 debug that did not die.
https://profiler.firefox.com/from-url/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FFg6CnG_zSDaYFtp_sErs8A%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_resource-usage.json/marker-chart/?globalTrackOrder=0&thread=0&timelineType=stack&v=10 is from a run on a ubuntu 18.04 worker.

Should we use workers with more ram, skip this test, change it so it's not so hungry, something else?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

is more ram == larger instance? right now the wayland machines are running on n2-standard-2 which is the same as t-linux-large* runs on. So we are running on the same instance size, I am not sure if the VM vs docker consumes more resources, nor if 2204/wayland consumes more than 1804/x11.

Comment 2

[Julien Cristau [:jcristau]]

They're both running out of memory, but the old worker seems to recover better than the new one.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

if we are going to make 2204 a real tier1, we need 2204-xlarge, so we should create that and give it a try

Comment 4

[Robert Longson [:longsonr]]

Attachment: Bug 1902979 - Don't dispatch SMIL events unless there are listeners r=smaug

Comment 5

[Robert Longson [:longsonr]]

https://treeherder.mozilla.org/jobs?repo=try&revision=671062535a77eb8aacc6910073bcd9acea5f07a1

Comment 6

[Phabricator Automation]

Comment on attachment 9407981 [details]
Bug 1902979 - Don't dispatch SMIL events unless there are listeners r=smaug

Revision D214010 was moved to bug 1903214. Setting attachment 9407981 [details] to obsolete.

Comment 7

[Robert Longson [:longsonr]]

bug 1903214 should help since we'll no longer send any events in these testcases.

We could also reduce the number of runnnables we create by having a queue of (target, event) pairs. And then a single runnable per animation frame tick or some such. That runnable would then dispatch many events. Or something along those lines.

Comment 8

[Julien Cristau [:jcristau]]

Looks like I misread the profiles yesterday. The ubuntu 18.04 workers for wpt have 16GB ram, while the wayland / 22.04 ones have only 8, so that explains why they behave differently. The test still times out, but that's separate from killing the worker.

Comment 9

[Julien Cristau [:jcristau]]

Attachment: Bug 1902979 - add t-linux-xlarge-wayland worker type. r?ahal,jmaher

This switches the wayland wpt tasks to run on xlarge workers, like the
corresponding x11 tasks.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f917602f4e57
add t-linux-xlarge-wayland worker type. r=jmaher,taskgraph-reviewers,bhearsum

Comment 11

[Ryan VanderMeulen [:RyanVM]]

This combined with bug 1903214 is looking great on autoland. Please request Beta approval when you get a chance.

Comment 12

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/f917602f4e57

Comment 13

[Julien Cristau [:jcristau]]

Attachment: Bug 1902979 - add t-linux-xlarge-wayland worker type. r?ahal,jmaher

This switches the wayland wpt tasks to run on xlarge workers, like the
corresponding x11 tasks.

Original Revision: https://phabricator.services.mozilla.com/D214127

Comment 14

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: none
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: test-only
• String changes made/needed: n/a
• Is Android affected?: no

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

is this not uplifting to beta? I saw the + and now the -;

Comment 16

[Ryan VanderMeulen [:RyanVM]]

See my comment in Phabricator.

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/fc8de7ce4388

Comment 18

[Intermittent Failures Robot]

1 failures in 4577 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• linux2204-64-wayland: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1902979&startday=2024-06-17&endday=2024-06-23&tree=all