<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

Comment 5

•

22 days ago

This looks to not be tree related, it was filed today and fails frequently but we have pushes from the 25th with this failures on retriggers here.
This points to either something in how the performance tests are set up or either something going on with the workers.

Flags: needinfo?(mcornmesser)

Flags: needinfo?(kshampur)

Whiteboard: [stockwell infra]

Comment 6

•

22 days ago

We had a change in the worker configuration earlier this week, https://github.com/mozilla-platform-ops/worker-images/pull/273 .

I will roll that pool back to the last config and deploy this afternoon.

Comment 7

•

22 days ago

The rollback config is going out now. Can you let me know if those tests come back green?

Flags: needinfo?(mcornmesser) → needinfo?(csabou)

Comment 8

•

22 days ago

Sure, will see how it goes.

Flags: needinfo?(csabou)

Comment 9

•

22 days ago

Still failing frequently, and we got two failures with https://treeherder.mozilla.org/logviewer?job_id=506249898&repo=autoland&lineNumber=428

Flags: needinfo?(mcornmesser)

Comment 10

•

22 days ago

It looks like we have this happening on 2 different pools. -hw and -hw-ref. I will roll back the config for the ref pool now.

Flags: needinfo?(mcornmesser) → needinfo?(csabou)

Comment 11

•

22 days ago

It looks like those tests are passing now. https://firefox-ci-tc.services.mozilla.com/tasks/ZzDm-RLYSQy4arxtCFzXuQ

jmoss: FYI

pmoore: This seems to have happened when we started using GW 83.5.7 on Windows 11 hardware workers. Not sure if it was the cause, but it was a significant difference between the 2 different configs.

jmaher: This may have been a condition that wasn't picked up by the integration testing.

Flags: needinfo?(pmoore)

Flags: needinfo?(jmoss)

Flags: needinfo?(jmaher)

Comment 12

•

22 days ago

Yes, can confirm lots of green runs on retriggers. Thanks for the fix.

Flags: needinfo?(csabou)

Updated

•

22 days ago

Flags: needinfo?(kshampur)

Comment hidden (Intermittent Failures Robot)

:jmoss

Comment 14

•

21 days ago

This is more than likely being caused by an update to generic worker and a feature called DisableOOMProtection (more here) in generic worker version 83.4.0. We are going to disable this workerconfig on the hardware pools as this feature impacts tasks such as this.

Flags: needinfo?(jmoss)

Greg Mierzwinski [:sparky]

Updated

•

21 days ago

Flags: needinfo?(jmaher)

Comment 15

•

21 days ago

Thanks for the quick resolution :markco, and :jmoss!

Updated

•

21 days ago

Flags: needinfo?(pmoore)

Comment hidden (Intermittent Failures Robot)

Comment 17

•

17 days ago

Hi all,

Please note, this feature was created to detect tasks that used excessive memory resources, and are at risk of failing intermittently due to limited available memory.

Disabling the feature may appear to improve the situation, but may hide deeper problems.

I would recommend at least investigating whether the tests can be adjusted to use less memory, rather than just disabling the worker protections.

Please note the excessive memory usage may originate from a long time ago, and not be related to a recent change.

If the current worker memory protections are too aggressive, there is also the opportunity to tune them. Please advise if this is desired. Thanks!

Comment 18

•

17 days ago

Hi Cosmin, Alexandru,

Can we make this bug a meta-bug to track the separate memory allocation issues?

It would be good to create dedicated bugs for each test suite/platform, and to NI the owners of these specific suites to these specific bugs (@jmaher can probably help identifying the folks).

Jonathan/Mark, are you happy to email out (to fx-all@m.o) with your plans for when the feature will be rolled out, and advise about this meta bug for reporting issues?

I think that should cover everything, so that moving forward we should hopefully have fewer intermittents due to excessive memory consumption.

Let me know if I can assist in any way.

Thanks!

Flags: needinfo?(mcornmesser)

Flags: needinfo?(jmoss)

Flags: needinfo?(csabou)

Flags: needinfo?(amarc)

Comment 19

•

17 days ago

•

Edited

Hi Pete, happy to make this a meta bug for memory allocation issues.
Want to double check, are we sure this is an issue with the tests rather than the workers? Latest failures here are all on the same worker R5CY128X71B and most of them on different test suites. Is only that worker that has this protective feature enabled on that platform, gecko-t-lambda-perf-a55? If not, it points to a worker issue rather than the test suite in my opinion.
This started as only tests running onwin11-64-24h2-hw-refand win11-64-24h2-hw were failing with task aborted due to sustained memory usage above 90%.

Alias: memory-allocation-issues

Component: Talos → General

Flags: needinfo?(pmoore)

Flags: needinfo?(csabou)

Flags: needinfo?(amarc)

Summary: Frequent [taskcluster:error] task aborted due to sustained memory usage above 90% → [meta] [please file individual bugs per test suite/platform] [taskcluster:error] task aborted due to sustained memory usage above 90%

Comment 20

•

17 days ago

Oh, this is a great call out. Indeed, that changes everything. This indeed suggests that particular worker is bad, and the feature does not need to be disabled. That makes our lives a lot simpler! It also provides a new insight into these issues - I don't think we had considered they might be localised to individual workers before, perhaps something else is running on the worker and eating up memory? Any ideas Mark or Jonathan?

Flags: needinfo?(pmoore)

Updated

•

17 days ago

Summary: [meta] [please file individual bugs per test suite/platform] [taskcluster:error] task aborted due to sustained memory usage above 90% → [meta] [taskcluster:error] task aborted due to sustained memory usage above 90%

:jmoss

Comment 21

•

17 days ago

Only releng-hardware/win11-64-24h2-hw-ref and releng-hardware/win11-64-24h2-hw have this generic worker feature disabled. All of the nodes here related to those 2 pools reflect all of the workers in the pools mentioned in releng-hardware/win11-64-24h2-hw-ref and releng-hardware/win11-64-24h2-hw. I believe the lambda perf pool is mobile phones.

Flags: needinfo?(jmoss)

Comment 22

•

17 days ago

if lambda stuff is all a specific worker id (i.e. R5CY128X71B), then we can quarantine/investigate/fix that worker. this is why we keep track of bugs in a big bucket and look for trends periodically.

Comment 23

•

17 days ago

t-nuc12-003 and t-nuc12-014 seem to hit this error more frequently

Comment 24

•

17 days ago

can we adjust this to be 95% memory instead of 90%?

BugBot (nomail) [:suhaib / :marco/ :calixte]

Comment 25

•

17 days ago

•

Edited

https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-t/worker-types/t-linux-docker/workers/us-west1-a/2489782782095831361?sortBy=started&sortDirection=desc
another one with problem, I've quarantined it for now.

Looks like it's hitting t-linux-docker workers; 2489782782095831361 and 7978773178479849871.

Updated

•

17 days ago

Keywords: meta

Julien Cristau [:jcristau]

Comment 26

•

17 days ago

ni Andrew for the mobile one - R5CY128X71B and for the t-linux-docker workers; 2489782782095831361 and 7978773178479849871.

Flags: needinfo?(aerickson)

:jmoss

Comment 27

•

17 days ago

Pinging :jcristau for the t-linux-docker workers.

Flags: needinfo?(jcristau)

Comment hidden (Intermittent Failures Robot)

Comment 29

•

16 days ago

(In reply to :jmoss from comment #27)

Pinging :jcristau for the t-linux-docker workers.

Seems we should disable the feature there.

Flags: needinfo?(jcristau)

Julien Cristau [:jcristau]

Comment 30

•

16 days ago

•

Edited

(In reply to Cosmin Sabou [:CosminS] from comment #25)

Looks like it's hitting t-linux-docker workers; 2489782782095831361 and 7978773178479849871.

Looking closer at https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-t/worker-types/t-linux-docker/workers/us-west1-a/2489782782095831361?sortBy=started&sortDirection=desc, it first happened for https://firefox-ci-tc.services.mozilla.com/tasks/HQPq-eEVRoKY7fq5xaOTYA/runs/0 (source-test-mozlint-android-focus). That does legitimately use up a lot of memory.
Then it hit following tasks again as they were just starting up. The reason for that is the container from HQPq-eEVRoKY7fq5xaOTYA was never actually killed so it was still running and using up memory in the background. That's similar to https://github.com/taskcluster/taskcluster/issues/7678, except not on timeout.

Comment 31

•

16 days ago

Created https://github.com/taskcluster/taskcluster/issues/7701