1984622 - Perma damp | netmonitor/custom.js: Test timed out | single tracking bug

Status: RESOLVED FIXED

(Testing :: Talos, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: chorotan [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=523573281&repo=mozilla-central&task=cb3IQHaTRzaL6shobpIMdw.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cb3IQHaTRzaL6shobpIMdw/runs/0/artifacts/public/logs/live_backing.log

---

task 2025-08-22T08:56:49.335+00:00] 08:56:49     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1399 / 45 requests
[task 2025-08-22T08:56:50.064+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1402 / 45 requests
[task 2025-08-22T08:56:50.079+00:00] 08:56:50     INFO -  PID 2494 | Received pageshow event for 77
[task 2025-08-22T08:56:50.079+00:00] 08:56:50     INFO -  PID 2494 | Wait for pending paints on 'custom.netmonitor.bigdatarequests.navigate'
[task 2025-08-22T08:56:50.079+00:00] 08:56:50     INFO -  PID 2494 | 'custom.netmonitor.bigdatarequests.navigate.settle.DAMP' took 0.06278399997972883ms.
[task 2025-08-22T08:56:50.445+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1405 / 45 requests
[task 2025-08-22T08:56:50.446+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1408 / 45 requests
[task 2025-08-22T08:56:50.446+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1411 / 45 requests
[task 2025-08-22T08:56:50.446+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1414 / 45 requests
[task 2025-08-22T08:56:50.446+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1417 / 45 requests
[task 2025-08-22T08:56:50.446+00:00] 08:56:50     INFO -  PID 2494 | [waitForAllRequestsFinished] Waiting for -1420 / 45 requests
[task 2025-08-22T09:01:44.489+00:00] 09:01:44     INFO -  PID 2494 | TEST-UNEXPECTED-FAIL | damp | netmonitor/custom.js: Test timed out
[task 2025-08-22T09:01:44.497+00:00] 09:01:44     INFO -  PID 2494 | [DampLoad helper] Unregister DampLoad actors
[task 2025-08-22T10:01:44.554+00:00] 10:01:44     INFO - Automation Error: mozharness timed out after 3600 seconds running ['/opt/worker/tasks/task_175585157930752/build/venv/bin/python', '/opt/worker/tasks/task_175585157930752/build/tests/talos/talos/run_tests.py', '--executablePath', '/opt/worker/tasks/task_175585157930752/build/application/Firefox Nightly.app/Contents/MacOS/firefox', '--suite', 'damp-other', '--title', 'macmini-r8-5', '--symbolsPath', 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JzUn1UQjT0qNLZQmHBLQCQ/artifacts/public/build/target.crashreporter-symbols.zip', '--project', 'mozilla-central', '--screenshot-on-failure', '--setpref=talos.damp.suite=other', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--log-tbpl-level=info', '--log-errorsummary=/opt/worker/tasks/task_175585157930752/build/blobber_upload_dir/damp-other_errorsummary.log']
[task 2025-08-22T10:01:44.558+00:00] 10:01:44     INFO - Return code: -9
[task 2025-08-22T10:01:44.558+00:00] 10:01:44  WARNING - setting return code to -9
[task 2025-08-22T10:01:44.559+00:00] 10:01:44  WARNING - setting return code to 2
[task 2025-08-22T10:01:44.559+00:00] 10:01:44     INFO - Running post-action listener: _package_coverage_data
[task 2025-08-22T10:01:44.559+00:00] 10:01:44     INFO - Running post-action listener: _resource_record_post_action
[task 2025-08-22T10:01:44.559+00:00] 10:01:44     INFO - Running post-action listener: process_java_coverage_data
[task 2025-08-22T10:01:44.559+00:00] 10:01:44     INFO - [mozharness: 2025-08-22 10:01:44.559316Z] Finished run-tests step (success)
[task 2025-08-22T10:01:44.559+00:00] 10:01:44     INFO - Running post-run listener: _resource_record_post_run
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - Total resource usage - Wall time: 5260s; CPU: 6%; Read bytes: 3850698752; Write bytes: 3710353408; Read time: 103739; Write time: 8742
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: CPU usage<br/>6.3%
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: I/O read bytes / time<br/>3,850,698,752 / 103,739
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: I/O write bytes / time<br/>3,710,353,408 / 8,742
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: CPU idle<br/>59,060.0 (93.7%)
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: CPU system<br/>1,667.6 (2.6%)
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: CPU user<br/>2,311.8 (3.7%)
[task 2025-08-22T10:01:47.423+00:00] 10:01:47     INFO - TinderboxPrint: Swap in / out<br/>806,707,200 / 0

Comment 1

[Cristina Horotan [:chorotan]]

Julian, can you please take a look?

Comment 2

[Julian Descottes [:jdescottes]]

Looking, it seems we are getting too many requests and never resolve in a specific test. I am suprised this would only regress on macos after Bug 1983496, not ruling out a regression from something else at this point.

I also seems to fail quite late in the test, and knowing that we run the same tests many time, this seems to indicate we hit an intermittent issue consistently enough to perma fail the job.

We can also see ~1500 requests finishing in this tests while we expect ~50. It doesn't really match the previous page (which has 2000 requests), but it surely doesn't match the page we are supposed to be on.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1983496

Comment 4

[Julian Descottes [:jdescottes]]

I have a few try pushes in progress, but since it's on mac, they might take a lot of time to get scheduled: https://treeherder.mozilla.org/jobs?repo=try&author=jdescottes%40mozilla.com&fromchange=0433dbbdc5670053a45f7434b0cd60b900ec4854&tochange=4c82e1a7cbc1cd977affdd9e863ccf3cd7b8af56

In the meantime I can't reproduce this locally.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

:bomsy, since you are the author of the regressor, bug 1983496, could you take a look?

For more information, please visit BugBot documentation.

Comment 6

[amarc]

Looks like it's green after backing out Bug 1830230: https://treeherder.mozilla.org/jobs?repo=autoland&duplicate_jobs=visible&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=95166f5b2a85db5652bc134b51d3d43a3455a26c&searchStr=macOS%2C14.70%2CShippable%2Copt%2CTalos%2Cperformance%2Ctests%2Ctest-macosx1470-64-shippable%2Fopt-talos-damp-other%2Cdamp-other&selectedTaskRun=Tm5tRQNQQGKgeoRKSF9fUA.0

Comment 7

[Intermittent Failures Robot]

10 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 5

Table

	no_variant
macosx1470-x86_64/opt	10

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1984622&startday=2025-08-18&endday=2025-08-24&tree=all