Frequent Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug
Categories
(Testing :: Talos, defect, P5)
Tracking
(firefox-esr128 unaffected, firefox137 unaffected, firefox138 affected, firefox139 affected)
| Tracking | Status | |
|---|---|---|
| firefox-esr128 | --- | unaffected |
| firefox137 | --- | unaffected |
| firefox138 | --- | affected |
| firefox139 | --- | affected |
People
(Reporter: intermittent-bug-filer, Assigned: sparky)
References
Details
(5 keywords, Whiteboard: [stockwell disable-recommended])
Attachments
(1 file)
Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=455604407&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F_5uNquESo-m4aItQ4JIUw/runs/0/artifacts/public/logs/live_backing.log
[task 2024-04-24T00:02:33.142Z] 00:02:33 INFO - TEST-INFO | 7696: exit 1
[task 2024-04-24T00:02:33.152Z] 00:02:33 INFO - mozcrash checking C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps for minidumps...
[task 2024-04-24T00:02:33.152Z] 00:02:33 INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/E7rAUrPmSLOI9XXwYbBx5w/artifacts/public/build/target.crashreporter-symbols.zip
[task 2024-04-24T00:02:47.241Z] 00:02:47 INFO - mozcrash Copy/paste: C:/task_171389853980003/fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_171389853980003\AppData\Local\Temp\tmpwnaidmo6\f80c181d-a18d-4f68-8e5a-121eefdb2c42.trace C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp C:\Users\task_171389853980003\AppData\Local\Temp\tmpo9o07qwk
[task 2024-04-24T00:02:47.335Z] 00:02:47 INFO - mozcrash Saved minidump as C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp
[task 2024-04-24T00:02:47.336Z] 00:02:47 INFO - mozcrash Saved app info as C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra
[task 2024-04-24T00:02:47.336Z] 00:02:47 INFO - PROCESS-CRASH | Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | glterrain
[task 2024-04-24T00:02:47.336Z] 00:02:47 INFO - Crash dump filename: C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp
[task 2024-04-24T00:02:47.337Z] 00:02:47 INFO - Process type: main
[task 2024-04-24T00:02:47.337Z] 00:02:47 INFO - Process pid: unknown
[task 2024-04-24T00:02:47.337Z] 00:02:47 INFO - Mozilla crash reason: Missing chrome or resource URLs: chrome://pageloader/content/utils.js
[task 2024-04-24T00:02:47.337Z] 00:02:47 INFO - minidump-stackwalk exited with return code 3221225595
[task 2024-04-24T00:02:47.337Z] 00:02:47 INFO - mozcrash Copy/paste: C:/task_171389853980003/fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_171389853980003\AppData\Local\Temp\tmpz1hkqpux\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.trace C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp C:\Users\task_171389853980003\AppData\Local\Temp\tmpo9o07qwk
[task 2024-04-24T00:02:47.343Z] 00:02:47 INFO - mozcrash Saved minidump as C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp
[task 2024-04-24T00:02:47.343Z] 00:02:47 INFO - mozcrash Saved app info as C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra
[task 2024-04-24T00:02:47.538Z] 00:02:47 INFO - PROCESS-CRASH | Missing chrome or resource URLs: resource://talos-powers/TalosContentProfiler.js [unknown top frame] | glterrain
[task 2024-04-24T00:02:47.538Z] 00:02:47 INFO - Crash dump filename: C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp
[task 2024-04-24T00:02:47.553Z] 00:02:47 INFO - Process type: content
[task 2024-04-24T00:02:47.553Z] 00:02:47 INFO - Process pid: unknown
[task 2024-04-24T00:02:47.553Z] 00:02:47 INFO - Mozilla crash reason: Missing chrome or resource URLs: resource://talos-powers/TalosContentProfiler.js
[task 2024-04-24T00:02:47.553Z] 00:02:47 INFO - minidump-stackwalk exited with return code 3221225595
[task 2024-04-24T00:02:47.553Z] 00:02:47 INFO - TEST-UNEXPECTED-ERROR | glterrain | Found crashes after test run, terminating test
[task 2024-04-24T00:02:47.554Z] 00:02:47 ERROR - Traceback (most recent call last):
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 250, in _runTest
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - test_results.add(
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 92, in add
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - browserLog = BrowserLogResults(
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - ^^^^^^^^^^^^^^^^^^
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 347, in __init__
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - self.parse()
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 373, in parse
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - self.error(
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 358, in error
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - raise utils.TalosError(message)
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - talos.utils.TalosError: Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))]
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - During handling of the above exception, another exception occurred:
[task 2024-04-24T00:02:47.554Z] 00:02:47 ERROR - Traceback (most recent call last):
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\run_tests.py", line 361, in run_tests
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - mytest.runTest(browser_config, test, utility_path=utility_path)
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 61, in runTest
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - return self._runTest(
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - ^^^^^^^^^^^^^^
[task 2024-04-24T00:02:47.554Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 261, in _runTest
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - self.check_for_crashes(
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 45, in check_for_crashes
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - raise TalosCrash("Found crashes after test run, terminating test")
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - talos.utils.TalosCrash: Found crashes after test run, terminating test
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - TEST-INFO took 26908ms
[task 2024-04-24T00:02:47.555Z] 00:02:47 INFO - SUITE-END | took 79s
[task 2024-04-24T00:02:47.835Z] 00:02:47 INFO - Return code: 2
[task 2024-04-24T00:02:47.835Z] 00:02:47 WARNING - setting return code to 2
[task 2024-04-24T00:02:47.839Z] 00:02:47 INFO - Running post-action listener: _package_coverage_data
[task 2024-04-24T00:02:47.839Z] 00:02:47 INFO - Running post-action listener: _resource_record_post_action
[task 2024-04-24T00:02:47.839Z] 00:02:47 INFO - Running post-action listener: process_java_coverage_data
[task 2024-04-24T00:02:47.839Z] 00:02:47 INFO - [mozharness: 2024-04-24 00:02:47.839472Z] Finished run-tests step (success)
[task 2024-04-24T00:02:47.839Z] 00:02:47 INFO - Running post-run listener: _resource_record_post_run
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - Total resource usage - Wall time: 81s; CPU: 4%; Read bytes: 377699840; Write bytes: 2008153088; Read time: 9; Write time: 2
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: CPU usage<br/>3.5%
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: I/O read bytes / time<br/>377,699,840 / 9
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: I/O write bytes / time<br/>2,008,153,088 / 2
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: CPU idle<br/>940.8 (96.4%)
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: CPU system<br/>20.4 (2.1%)
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - TinderboxPrint: CPU user<br/>13.1 (1.3%)
[task 2024-04-24T00:02:47.913Z] 00:02:47 INFO - install - Wall time: 2s; CPU: 6%; Read bytes: 1742848; Write bytes: 4648960; Read time: 0; Write time: 0
[task 2024-04-24T00:02:47.920Z] 00:02:47 INFO - run-tests - Wall time: 80s; CPU: 3%; Read bytes: 375956992; Write bytes: 2003504128; Read time: 9; Write time: 2
[task 2024-04-24T00:02:48.085Z] 00:02:48 WARNING - returning nonzero exit status 2
[taskcluster 2024-04-24T00:02:48.116Z] Exit Code: 2
[taskcluster 2024-04-24T00:02:48.116Z] User Time: 0s
[taskcluster 2024-04-24T00:02:48.116Z] Kernel Time: 15.625ms
[taskcluster 2024-04-24T00:02:48.116Z] Wall Time: 2m28.4157583s
[taskcluster 2024-04-24T00:02:48.116Z] Result: FAILED
[taskcluster 2024-04-24T00:02:48.116Z] === Task Finished ===
[taskcluster 2024-04-24T00:02:48.116Z] Task Duration: 2m28.4168333s
[taskcluster 2024-04-24T00:02:48.975Z] Uploading artifact public/logs/localconfig.json from file C:\task_171389853980003\logs\localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.290Z] Uploading artifact public/test_info/b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp from file C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.588Z] Uploading artifact public/test_info/b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra from file C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.886Z] Uploading artifact public/test_info/f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp from file C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.230Z] Uploading artifact public/test_info/f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra from file C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.501Z] Uploading artifact public/test_info/profile_resource-usage.json from file C:\task_171389853980003\build\blobber_upload_dir\profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.782Z] Uploading artifact public/test_info/resource-usage.json from file C:\task_171389853980003\build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:51.112Z] Uploading artifact public/test_info/webgl_errorsummary.log from file C:\task_171389853980003\build\blobber_upload_dir\webgl_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:51.379Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-07-22T21:44:45.259Z
[taskcluster:error] exit status 2
| Comment hidden (Intermittent Failures Robot) |
Comment 2•1 year ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.
| Reporter | ||
Comment 3•1 year ago
|
||
| treeherder | ||
New failure instance: https://treeherder.mozilla.org/logviewer?job_id=464783628&repo=autoland
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 41•8 months ago
|
||
tier 1 failure https://treeherder.mozilla.org/logviewer?job_id=497605234&repo=autoland&lineNumber=1234
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 46•7 months ago
|
||
See https://bugzilla.mozilla.org/show_bug.cgi?id=1915321#c30 for context on the spike in failures here.
| Comment hidden (Intermittent Failures Robot) |
Updated•6 months ago
|
Comment 49•6 months ago
|
||
Set release status flags based on info from the regressing bug 1956329
:atossou, since you are the author of the regressor, bug 1956329, could you take a look?
For more information, please visit BugBot documentation.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 66•5 months ago
|
||
Here is a treeherder run https://treeherder.mozilla.org/jobs?repo=try&revision=f4189bfe238972981088185e7f9acfcf75dfd9ac&selectedTaskRun=LQ3HEIozRx-2vR0Z5PEGPQ.0 demonstrating that reverting the patch from https://phabricator.services.mozilla.com/D242989 does not address the bug here. In particular the bug is still present even after reverting the patch.
So https://bugzilla.mozilla.org/show_bug.cgi?id=1956329 is not the cause of this bug
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 70•5 months ago
|
||
:aglavic, could you help us assign this to someone? Thanks
See also Bug 1915321
| Comment hidden (Intermittent Failures Robot) |
Comment 72•5 months ago
|
||
You're right, this bug is related to the failures seen in Bug 1915321, that bug is being worked on by :fbilt
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 81•4 months ago
|
||
Hi mconley!
Could you check the logs and dump/extra files to see if you notice anything unusual?
https://treeherder.mozilla.org/jobs?repo=autoland&revision=a5c30edd48c8babbe4cd5b012dc346948332c22c&selectedTaskRun=F1SUKdHMRcCCJgE6j9ibzw.0&searchStr=Windows%2C11%2C24H2%2CShippable%2Copt%2CTalos%2Cperformance%2Ctests%2Ctest-windows11-64-24h2-shippable%2Fopt-talos-tp5o%2Ctp
Comment 82•4 months ago
|
||
What's interesting is that the crash / access attempt occurs after the test has run, where we clearly see the pageloader script logging things earlier on (like from this).
That makes me think what has happened is that this chrome:// URI access is occurring after the pageloader extension shuts down, and after it deregisters its chrome handle here.
There's an event handler registered here that accesses that chrome URI: https://searchfox.org/mozilla-central/rev/61f64da2b20e50610d532675877ca4b5b9d6f1e4/testing/talos/talos/pageloader/chrome/pageloader.js#245
I don't see that event handler ever being unregistered. My guess is that we sometimes have a race, where a remoteness flip event is firing just after the extension shuts down, and that's causing the crash. My recommended solution is to make this event handler removable, and removing it in the plStopAll routine - probably in here somewhere: https://searchfox.org/mozilla-central/rev/61f64da2b20e50610d532675877ca4b5b9d6f1e4/testing/talos/talos/pageloader/chrome/pageloader.js#904-930
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 90•4 months ago
|
||
| Comment hidden (Intermittent Failures Robot) |
Comment 92•3 months ago
|
||
(In reply to Mike Conley (:mconley) (:⚙️) from comment #90)
https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=139894
hi Mike! seems like your patch fixes things? mind pushing it for a review?
Comment 93•3 months ago
|
||
I'm afraid it didn't - there are two instances of the failure / crash in that push.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 110•2 months ago
|
||
Hi
Do you see anything unusual in this try push in the logs, or dmp/extra files?
It seems that the issue originates from the following line, which causes the "Exiting due to channel error" message (link log). This appears to happen because the tab isn't fully initialized when we attempt to inject the JavaScript frame scripts.
As a potential fix, I tried adding a short delay (~200ms) before performing the injection. However, that leads to another issue:
FailTimeout (3/3) exceeded on ... link to error/log
It seems like there's a timeout defined here. I attempted to increase that timeout, but it didn't appear to have any effect.
Looking more closely at the logs, the process seems to hang right after the JS files are injected, but I’m not sure why. There are no useful errors or exceptions thrown that could help trace it further.
Almost fixed in this try run (delay ~75ms )
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Assignee | ||
Comment 113•2 months ago
|
||
To add to what :fbilt mentions, something that's interesting in the logs is that when the page hangs, the TabRemotenessChange event gets continuously called: https://searchfox.org/mozilla-central/source/testing/talos/talos/pageloader/chrome/pageloader.js#245
On passing runs, it only gets called once.
:fbilt have you tried to increase the maxRetries? https://searchfox.org/mozilla-central/source/testing/talos/talos/pageloader/chrome/pageloader.js#21
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 118•2 months ago
|
||
try push with 75 ms delay and maxRetries = 15.
try push with 100 ms delay and maxRetries = 15.
try push with 200 ms delay and maxRetries = 15.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 125•2 months ago
|
||
I'm afraid I don't have any update here. I'm not sure why this is happening, nor how to fix it.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Assignee | ||
Comment 131•1 month ago
|
||
Found a bug in the screen resolution setting and was hoping that it might be causing this issue (bug 1986969), but fixing that didn't make an impact: https://treeherder.mozilla.org/jobs?repo=try&revision=8da352ee47ee3912766268277f49425b987df728
| Assignee | ||
Comment 132•1 month ago
|
||
So regarding all those addMsdListener calls that :fbilt found, they are very likely from repeatably calling this on each page without removing the existing TabRemotenessChange listener: https://searchfox.org/firefox-main/source/testing/talos/talos/pageloader/chrome/pageloader.js#303
I think :mconley was on the right track in comment #82. There's definitely a race condition happening here that prevents us from removing the message listeners. I wasn't doing this in the same way that :mconley's patch did, but I tried removing the message listener just before adding it again. Talos completely broke when that happened and it seems to be because we are waiting for this field to be set: https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#53
I'm still digging into this so that info may change. So far it seems like Talos depends on having multiple listeners on the TabRemotenessChange event because of this race-condition-like issue.
Something I noticed in :mconley's patch is that the ContentListener listener setup isn't being removed so that might be something to explore.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Assignee | ||
Comment 135•1 month ago
|
||
So I had a bit of promising run here 2 failures in about 100 runs: https://treeherder.mozilla.org/jobs?repo=try&revision=75eaaa1a8fd4b5226f1d6d94e32eaebe0bbb0d10
Only thing I added there was the removal of the ContentListener listeners. I'll try that with :mconley's patch now.
| Assignee | ||
Comment 136•1 month ago
|
||
Testing locally, and it looks like the patch that :mconley provided isn't removing the listener that triggers addMsgListeners so as the test run continues, the number of times it gets called continuously increases.
| Assignee | ||
Comment 137•1 month ago
|
||
It looks like this code: https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#302-306
Should be moved to here (after the loadURI calls): https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#405
That resolves the race-like condition that I see locally that prevents me from removing the msg listeners in its original location.
| Assignee | ||
Comment 138•1 month ago
|
||
New try run with the changes from comment #137: https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=151916
| Assignee | ||
Comment 139•1 month ago
|
||
New try run here: https://treeherder.mozilla.org/jobs?repo=try&revision=04915b46b3df154ee3dacc3147ce74839da13795
So combining :mconley's changes with my own, as well as moving the addMsgListeners call I was able to get rid of the race-like condition and remove the previous message listeners before re-adding them. However, it did not resolve this bug.
I ran another try run where I output the contents of the profile extensions directory here:
It looks like something is deleting the profile extension directory contents: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=ARperibHTsyh88rHYAkTDw.0&revision=404dd9a901c8ed64f0e9e68d4743af0a1ef66014
[task 2025-09-08T21:39:32.265+00:00] 21:39:32 INFO - -rw-rw-rw- FILE api.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - drwxrwxrwx DIR chrome
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE manifest.json
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE README
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE schema.json
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE a11y.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE lh_dummy.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32 INFO - -rw-rw-rw- FILE lh_fnbpaint.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE lh_hero.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE lh_moz.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE lh_pdfpaint.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE MozillaFileLogger.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE pageloader.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE pageloader.xhtml
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE Profiler.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE quit.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE report.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32 INFO - -rw-rw-rw- FILE talos-content.js
[task 2025-09-08T21:39:32.268+00:00] 21:39:32 INFO - -rw-rw-rw- FILE tscroll.js
[task 2025-09-08T21:39:32.268+00:00] 21:39:32 INFO - -rw-rw-rw- FILE utils.js
[task 2025-09-08T21:39:32.576+00:00] 21:39:32 INFO - PID 11020 | Cycle 1(23): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:33.093+00:00] 21:39:33 INFO - PID 11020 | Cycle 1(24): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:33.268+00:00] 21:39:33 INFO - drwxrwxrwx DIR chrome
[task 2025-09-08T21:39:33.628+00:00] 21:39:33 INFO - PID 11020 | Cycle 1(25): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:34.124+00:00] 21:39:34 INFO - PID 11020 | remoteness changed?
[task 2025-09-08T21:39:34.124+00:00] 21:39:34 INFO - PID 11020 | called
[task 2025-09-08T21:39:34.269+00:00] 21:39:34 INFO - drwxrwxrwx DIR chrome
| Assignee | ||
Comment 140•1 month ago
|
||
Ok, so I think I've found the issue that's causing this. In this try run, I moved the profile location to another folder and I didn't get the failure in 100 runs of tp5o: https://treeherder.mozilla.org/jobs?repo=try&revision=11c12008c3b2a76a3e6446bb5f0a4b8fe2979a61
I believe the issue is that since the we store our profile in the location of LOCALAPPDATA, windows is deleting the folder part way through our test runs. The profile path is usually: C:\Users\task_XYZ\AppData\Local\Temp\tmpib6mqhi2\profile
LOCALAPPDATA points to C:\Users\task_XYZ\AppData\Local\Temp\. Also, we get this location through a tempfile call.
| Assignee | ||
Comment 141•1 month ago
|
||
I spoke with :jmoss on slack about this, and he's going to look into getting Storage Sense disabled on the windows hardware (we think that's what is causing this issue). See this relops ticket: https://mozilla-hub.atlassian.net/browse/RELOPS-1864
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 150•1 month ago
|
||
:jmoss have you had a chance to look into this frequent failure? We already have 84 total failures and 382 total failures in the last 30 days.
Updated•1 month ago
|
Comment 151•1 month ago
|
||
(In reply to Cristina Horotan [:chorotan] from comment #150)
:jmoss have you had a chance to look into this frequent failure? We already have 84 total failures and 382 total failures in the last 30 days.
We plan to address this next week. I will confirm with :sparky once deployed.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Assignee | ||
Comment 158•24 days ago
|
||
This patch fixes an issue in talos where the profile directory gets deleted partway through a test. It changes the location of the profile directory from the temporary location (e.g. LOCALAPPDATA for windows) to TASK_WORKDIR. The profile directory is still cleaned up at the end of the test run as well.
Updated•24 days ago
|
| Assignee | ||
Updated•24 days ago
|
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Comment 161•16 days ago
|
||
Comment 162•16 days ago
|
||
| bugherder | ||
| Comment hidden (Intermittent Failures Robot) |
Comment 164•2 days ago
|
||
(In reply to Pulsebot from comment #161)
Pushed by gmierz2@outlook.com:
https://github.com/mozilla-firefox/firefox/commit/b60c123daa3f
https://hg.mozilla.org/integration/autoland/rev/4d7843cb7023
Change browser profile location to TASK_WORKDIR in CI.
r=perftest-reviewers,aglavic
Perfherder has detected a talos performance change from push 4d7843cb7023354de9e7e1fc438b7ae220ec317d.
If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.
Improvements:
| Ratio | Test | Platform | Options | Absolute values (old vs new) |
|---|---|---|---|---|
| 3% | tp5n nonmain_normal_fileio | windows11-64-24h2-shippable | e10s fission stylo webrender-sw | 536,271,581.08 -> 519,213,557.42 |
Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.
If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.
You can run all of these tests on try with ./mach try perf --alert 47195
The following documentation link provides more information about this command.
Updated•2 days ago
|
Description
•