Open Bug 1893092 Opened 1 year ago Updated 2 days ago

Frequent Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug

Categories

(Testing :: Talos, defect, P5)

defect

Tracking

(firefox-esr128 unaffected, firefox137 unaffected, firefox138 affected, firefox139 affected)

REOPENED
Tracking Status
firefox-esr128 --- unaffected
firefox137 --- unaffected
firefox138 --- affected
firefox139 --- affected

People

(Reporter: intermittent-bug-filer, Assigned: sparky)

References

Details

(5 keywords, Whiteboard: [stockwell disable-recommended])

Attachments

(1 file)

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=455604407&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F_5uNquESo-m4aItQ4JIUw/runs/0/artifacts/public/logs/live_backing.log


[task 2024-04-24T00:02:33.142Z] 00:02:33     INFO -  TEST-INFO | 7696: exit 1
[task 2024-04-24T00:02:33.152Z] 00:02:33     INFO -  mozcrash checking C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps for minidumps...
[task 2024-04-24T00:02:33.152Z] 00:02:33     INFO -  mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/E7rAUrPmSLOI9XXwYbBx5w/artifacts/public/build/target.crashreporter-symbols.zip
[task 2024-04-24T00:02:47.241Z] 00:02:47     INFO -  mozcrash Copy/paste: C:/task_171389853980003/fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_171389853980003\AppData\Local\Temp\tmpwnaidmo6\f80c181d-a18d-4f68-8e5a-121eefdb2c42.trace C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp C:\Users\task_171389853980003\AppData\Local\Temp\tmpo9o07qwk
[task 2024-04-24T00:02:47.335Z] 00:02:47     INFO -  mozcrash Saved minidump as C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp
[task 2024-04-24T00:02:47.336Z] 00:02:47     INFO -  mozcrash Saved app info as C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra
[task 2024-04-24T00:02:47.336Z] 00:02:47     INFO -  PROCESS-CRASH | Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | glterrain
[task 2024-04-24T00:02:47.336Z] 00:02:47     INFO -  Crash dump filename: C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp
[task 2024-04-24T00:02:47.337Z] 00:02:47     INFO -  Process type: main
[task 2024-04-24T00:02:47.337Z] 00:02:47     INFO -  Process pid: unknown
[task 2024-04-24T00:02:47.337Z] 00:02:47     INFO -  Mozilla crash reason: Missing chrome or resource URLs: chrome://pageloader/content/utils.js
[task 2024-04-24T00:02:47.337Z] 00:02:47     INFO -  minidump-stackwalk exited with return code 3221225595
[task 2024-04-24T00:02:47.337Z] 00:02:47     INFO -  mozcrash Copy/paste: C:/task_171389853980003/fetches\minidump-stackwalk\minidump-stackwalk.exe --symbols-url=https://symbols.mozilla.org/ --cyborg=C:\Users\task_171389853980003\AppData\Local\Temp\tmpz1hkqpux\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.trace C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp C:\Users\task_171389853980003\AppData\Local\Temp\tmpo9o07qwk
[task 2024-04-24T00:02:47.343Z] 00:02:47     INFO -  mozcrash Saved minidump as C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp
[task 2024-04-24T00:02:47.343Z] 00:02:47     INFO -  mozcrash Saved app info as C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra
[task 2024-04-24T00:02:47.538Z] 00:02:47     INFO -  PROCESS-CRASH | Missing chrome or resource URLs: resource://talos-powers/TalosContentProfiler.js [unknown top frame] | glterrain
[task 2024-04-24T00:02:47.538Z] 00:02:47     INFO -  Crash dump filename: C:\Users\task_171389853980003\AppData\Local\Temp\tmp276_z3w4\profile\minidumps\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp
[task 2024-04-24T00:02:47.553Z] 00:02:47     INFO -  Process type: content
[task 2024-04-24T00:02:47.553Z] 00:02:47     INFO -  Process pid: unknown
[task 2024-04-24T00:02:47.553Z] 00:02:47     INFO -  Mozilla crash reason: Missing chrome or resource URLs: resource://talos-powers/TalosContentProfiler.js
[task 2024-04-24T00:02:47.553Z] 00:02:47     INFO -  minidump-stackwalk exited with return code 3221225595
[task 2024-04-24T00:02:47.553Z] 00:02:47     INFO -  TEST-UNEXPECTED-ERROR | glterrain | Found crashes after test run, terminating test
[task 2024-04-24T00:02:47.554Z] 00:02:47    ERROR -  Traceback (most recent call last):
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 250, in _runTest
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      test_results.add(
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 92, in add
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      browserLog = BrowserLogResults(
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -                   ^^^^^^^^^^^^^^^^^^
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 347, in __init__
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      self.parse()
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 373, in parse
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      self.error(
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\results.py", line 358, in error
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      raise utils.TalosError(message)
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -  talos.utils.TalosError: Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))]
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -  During handling of the above exception, another exception occurred:
[task 2024-04-24T00:02:47.554Z] 00:02:47    ERROR -  Traceback (most recent call last):
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\run_tests.py", line 361, in run_tests
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      mytest.runTest(browser_config, test, utility_path=utility_path)
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 61, in runTest
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -      return self._runTest(
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -             ^^^^^^^^^^^^^^
[task 2024-04-24T00:02:47.554Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 261, in _runTest
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -      self.check_for_crashes(
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -    File "C:\task_171389853980003\build\tests\talos\talos\ttest.py", line 45, in check_for_crashes
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -      raise TalosCrash("Found crashes after test run, terminating test")
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -  talos.utils.TalosCrash: Found crashes after test run, terminating test
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -  TEST-INFO took 26908ms
[task 2024-04-24T00:02:47.555Z] 00:02:47     INFO -  SUITE-END | took 79s
[task 2024-04-24T00:02:47.835Z] 00:02:47     INFO - Return code: 2
[task 2024-04-24T00:02:47.835Z] 00:02:47  WARNING - setting return code to 2
[task 2024-04-24T00:02:47.839Z] 00:02:47     INFO - Running post-action listener: _package_coverage_data
[task 2024-04-24T00:02:47.839Z] 00:02:47     INFO - Running post-action listener: _resource_record_post_action
[task 2024-04-24T00:02:47.839Z] 00:02:47     INFO - Running post-action listener: process_java_coverage_data
[task 2024-04-24T00:02:47.839Z] 00:02:47     INFO - [mozharness: 2024-04-24 00:02:47.839472Z] Finished run-tests step (success)
[task 2024-04-24T00:02:47.839Z] 00:02:47     INFO - Running post-run listener: _resource_record_post_run
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - Total resource usage - Wall time: 81s; CPU: 4%; Read bytes: 377699840; Write bytes: 2008153088; Read time: 9; Write time: 2
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: CPU usage<br/>3.5%
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: I/O read bytes / time<br/>377,699,840 / 9
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: I/O write bytes / time<br/>2,008,153,088 / 2
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: CPU idle<br/>940.8 (96.4%)
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: CPU system<br/>20.4 (2.1%)
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - TinderboxPrint: CPU user<br/>13.1 (1.3%)
[task 2024-04-24T00:02:47.913Z] 00:02:47     INFO - install - Wall time: 2s; CPU: 6%; Read bytes: 1742848; Write bytes: 4648960; Read time: 0; Write time: 0
[task 2024-04-24T00:02:47.920Z] 00:02:47     INFO - run-tests - Wall time: 80s; CPU: 3%; Read bytes: 375956992; Write bytes: 2003504128; Read time: 9; Write time: 2
[task 2024-04-24T00:02:48.085Z] 00:02:48  WARNING - returning nonzero exit status 2
[taskcluster 2024-04-24T00:02:48.116Z]    Exit Code: 2
[taskcluster 2024-04-24T00:02:48.116Z]    User Time: 0s
[taskcluster 2024-04-24T00:02:48.116Z]  Kernel Time: 15.625ms
[taskcluster 2024-04-24T00:02:48.116Z]    Wall Time: 2m28.4157583s
[taskcluster 2024-04-24T00:02:48.116Z]       Result: FAILED
[taskcluster 2024-04-24T00:02:48.116Z] === Task Finished ===
[taskcluster 2024-04-24T00:02:48.116Z] Task Duration: 2m28.4168333s
[taskcluster 2024-04-24T00:02:48.975Z] Uploading artifact public/logs/localconfig.json from file C:\task_171389853980003\logs\localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.290Z] Uploading artifact public/test_info/b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp from file C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.dmp with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.588Z] Uploading artifact public/test_info/b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra from file C:\task_171389853980003\build\blobber_upload_dir\b7197668-2cf9-4834-9bd2-a4b70fef4bd8.extra with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:49.886Z] Uploading artifact public/test_info/f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp from file C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.dmp with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.230Z] Uploading artifact public/test_info/f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra from file C:\task_171389853980003\build\blobber_upload_dir\f80c181d-a18d-4f68-8e5a-121eefdb2c42.extra with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.501Z] Uploading artifact public/test_info/profile_resource-usage.json from file C:\task_171389853980003\build\blobber_upload_dir\profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:50.782Z] Uploading artifact public/test_info/resource-usage.json from file C:\task_171389853980003\build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:51.112Z] Uploading artifact public/test_info/webgl_errorsummary.log from file C:\task_171389853980003\build\blobber_upload_dir\webgl_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2024-07-22T21:44:45.259Z
[taskcluster 2024-04-24T00:02:51.379Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-07-22T21:44:45.259Z
[taskcluster:error] exit status 2
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Summary: Intermittent [tier 2] Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug → Intermittent Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug

See https://bugzilla.mozilla.org/show_bug.cgi?id=1915321#c30 for context on the spike in failures here.

Duplicate of this bug: 1958662
Keywords: regression
Regressed by: 1956329
Summary: Intermittent Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug → Frequent Missing chrome or resource URLs: chrome://pageloader/content/utils.js [unknown top frame] | single tracking bug

Set release status flags based on info from the regressing bug 1956329

:atossou, since you are the author of the regressor, bug 1956329, could you take a look?

For more information, please visit BugBot documentation.

Here is a treeherder run https://treeherder.mozilla.org/jobs?repo=try&revision=f4189bfe238972981088185e7f9acfcf75dfd9ac&selectedTaskRun=LQ3HEIozRx-2vR0Z5PEGPQ.0 demonstrating that reverting the patch from https://phabricator.services.mozilla.com/D242989 does not address the bug here. In particular the bug is still present even after reverting the patch.

So https://bugzilla.mozilla.org/show_bug.cgi?id=1956329 is not the cause of this bug

Flags: needinfo?(atossou)

:aglavic, could you help us assign this to someone? Thanks
See also Bug 1915321

Flags: needinfo?(aglavic)
No longer regressed by: 1956329
See Also: → 1915321

You're right, this bug is related to the failures seen in Bug 1915321, that bug is being worked on by :fbilt

Flags: needinfo?(aglavic)

What's interesting is that the crash / access attempt occurs after the test has run, where we clearly see the pageloader script logging things earlier on (like from this).

That makes me think what has happened is that this chrome:// URI access is occurring after the pageloader extension shuts down, and after it deregisters its chrome handle here.

There's an event handler registered here that accesses that chrome URI: https://searchfox.org/mozilla-central/rev/61f64da2b20e50610d532675877ca4b5b9d6f1e4/testing/talos/talos/pageloader/chrome/pageloader.js#245

I don't see that event handler ever being unregistered. My guess is that we sometimes have a race, where a remoteness flip event is firing just after the extension shuts down, and that's causing the crash. My recommended solution is to make this event handler removable, and removing it in the plStopAll routine - probably in here somewhere: https://searchfox.org/mozilla-central/rev/61f64da2b20e50610d532675877ca4b5b9d6f1e4/testing/talos/talos/pageloader/chrome/pageloader.js#904-930

Flags: needinfo?(mconley)

(In reply to Mike Conley (:mconley) (:⚙️) from comment #90)

https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=139894

hi Mike! seems like your patch fixes things? mind pushing it for a review?

Flags: needinfo?(mconley)

I'm afraid it didn't - there are two instances of the failure / crash in that push.

Flags: needinfo?(mconley)
Whiteboard: [stockwell disable-recommended]

Hi
Do you see anything unusual in this try push in the logs, or dmp/extra files?

It seems that the issue originates from the following line, which causes the "Exiting due to channel error" message (link log). This appears to happen because the tab isn't fully initialized when we attempt to inject the JavaScript frame scripts.
As a potential fix, I tried adding a short delay (~200ms) before performing the injection. However, that leads to another issue:
FailTimeout (3/3) exceeded on ... link to error/log
It seems like there's a timeout defined here. I attempted to increase that timeout, but it didn't appear to have any effect.
Looking more closely at the logs, the process seems to hang right after the JS files are injected, but I’m not sure why. There are no useful errors or exceptions thrown that could help trace it further.

Almost fixed in this try run (delay ~75ms )

Flags: needinfo?(mconley)

To add to what :fbilt mentions, something that's interesting in the logs is that when the page hangs, the TabRemotenessChange event gets continuously called: https://searchfox.org/mozilla-central/source/testing/talos/talos/pageloader/chrome/pageloader.js#245

On passing runs, it only gets called once.

:fbilt have you tried to increase the maxRetries? https://searchfox.org/mozilla-central/source/testing/talos/talos/pageloader/chrome/pageloader.js#21

Flags: needinfo?(fbilt)

try push with 75 ms delay and maxRetries = 15.
try push with 100 ms delay and maxRetries = 15.
try push with 200 ms delay and maxRetries = 15.

Flags: needinfo?(fbilt)

I'm afraid I don't have any update here. I'm not sure why this is happening, nor how to fix it.

Flags: needinfo?(mconley)

Found a bug in the screen resolution setting and was hoping that it might be causing this issue (bug 1986969), but fixing that didn't make an impact: https://treeherder.mozilla.org/jobs?repo=try&revision=8da352ee47ee3912766268277f49425b987df728

So regarding all those addMsdListener calls that :fbilt found, they are very likely from repeatably calling this on each page without removing the existing TabRemotenessChange listener: https://searchfox.org/firefox-main/source/testing/talos/talos/pageloader/chrome/pageloader.js#303

I think :mconley was on the right track in comment #82. There's definitely a race condition happening here that prevents us from removing the message listeners. I wasn't doing this in the same way that :mconley's patch did, but I tried removing the message listener just before adding it again. Talos completely broke when that happened and it seems to be because we are waiting for this field to be set: https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#53

I'm still digging into this so that info may change. So far it seems like Talos depends on having multiple listeners on the TabRemotenessChange event because of this race-condition-like issue.

Something I noticed in :mconley's patch is that the ContentListener listener setup isn't being removed so that might be something to explore.

So I had a bit of promising run here 2 failures in about 100 runs: https://treeherder.mozilla.org/jobs?repo=try&revision=75eaaa1a8fd4b5226f1d6d94e32eaebe0bbb0d10

Only thing I added there was the removal of the ContentListener listeners. I'll try that with :mconley's patch now.

Testing locally, and it looks like the patch that :mconley provided isn't removing the listener that triggers addMsgListeners so as the test run continues, the number of times it gets called continuously increases.

It looks like this code: https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#302-306

Should be moved to here (after the loadURI calls): https://searchfox.org/firefox-main/rev/010c6878a1520b36721b63846d1b0c44b23524d0/testing/talos/talos/pageloader/chrome/pageloader.js#405

That resolves the race-like condition that I see locally that prevents me from removing the msg listeners in its original location.

New try run here: https://treeherder.mozilla.org/jobs?repo=try&revision=04915b46b3df154ee3dacc3147ce74839da13795

So combining :mconley's changes with my own, as well as moving the addMsgListeners call I was able to get rid of the race-like condition and remove the previous message listeners before re-adding them. However, it did not resolve this bug.

I ran another try run where I output the contents of the profile extensions directory here:

It looks like something is deleting the profile extension directory contents: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=ARperibHTsyh88rHYAkTDw.0&revision=404dd9a901c8ed64f0e9e68d4743af0a1ef66014

[task 2025-09-08T21:39:32.265+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  api.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  drwxrwxrwx  DIR   chrome
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  manifest.json
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  README
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  schema.json
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  a11y.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  lh_dummy.js
[task 2025-09-08T21:39:32.266+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  lh_fnbpaint.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  lh_hero.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  lh_moz.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  lh_pdfpaint.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  MozillaFileLogger.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  pageloader.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  pageloader.xhtml
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  Profiler.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  quit.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  report.js
[task 2025-09-08T21:39:32.267+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  talos-content.js
[task 2025-09-08T21:39:32.268+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  tscroll.js
[task 2025-09-08T21:39:32.268+00:00] 21:39:32     INFO -  -rw-rw-rw-  FILE  utils.js
[task 2025-09-08T21:39:32.576+00:00] 21:39:32     INFO -  PID 11020 | Cycle 1(23): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:33.093+00:00] 21:39:33     INFO -  PID 11020 | Cycle 1(24): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:33.268+00:00] 21:39:33     INFO -  drwxrwxrwx  DIR   chrome
[task 2025-09-08T21:39:33.628+00:00] 21:39:33     INFO -  PID 11020 | Cycle 1(25): loaded http://google.com-talos/tests/tp5n/google.com/www.google.com/search@q=mozilla.html (next: http://goo.ne.jp-talos/tests/tp5n/goo.ne.jp/goo.ne.jp/index.html)
[task 2025-09-08T21:39:34.124+00:00] 21:39:34     INFO -  PID 11020 | remoteness changed?
[task 2025-09-08T21:39:34.124+00:00] 21:39:34     INFO -  PID 11020 | called
[task 2025-09-08T21:39:34.269+00:00] 21:39:34     INFO -  drwxrwxrwx  DIR   chrome
See Also: → 1987552

Ok, so I think I've found the issue that's causing this. In this try run, I moved the profile location to another folder and I didn't get the failure in 100 runs of tp5o: https://treeherder.mozilla.org/jobs?repo=try&revision=11c12008c3b2a76a3e6446bb5f0a4b8fe2979a61

I believe the issue is that since the we store our profile in the location of LOCALAPPDATA, windows is deleting the folder part way through our test runs. The profile path is usually: C:\Users\task_XYZ\AppData\Local\Temp\tmpib6mqhi2\profile

LOCALAPPDATA points to C:\Users\task_XYZ\AppData\Local\Temp\. Also, we get this location through a tempfile call.

I spoke with :jmoss on slack about this, and he's going to look into getting Storage Sense disabled on the windows hardware (we think that's what is causing this issue). See this relops ticket: https://mozilla-hub.atlassian.net/browse/RELOPS-1864

:jmoss have you had a chance to look into this frequent failure? We already have 84 total failures and 382 total failures in the last 30 days.

Flags: needinfo?(jmoss)
Whiteboard: [stockwell disable-recommended]

(In reply to Cristina Horotan [:chorotan] from comment #150)

:jmoss have you had a chance to look into this frequent failure? We already have 84 total failures and 382 total failures in the last 30 days.

We plan to address this next week. I will confirm with :sparky once deployed.

Flags: needinfo?(jmoss)

This patch fixes an issue in talos where the profile directory gets deleted partway through a test. It changes the location of the profile directory from the temporary location (e.g. LOCALAPPDATA for windows) to TASK_WORKDIR. The profile directory is still cleaned up at the end of the test run as well.

Assignee: nobody → gmierzwinski
Keywords: leave-open
Pushed by gmierz2@outlook.com: https://github.com/mozilla-firefox/firefox/commit/b60c123daa3f https://hg.mozilla.org/integration/autoland/rev/4d7843cb7023 Change browser profile location to TASK_WORKDIR in CI. r=perftest-reviewers,aglavic

(In reply to Pulsebot from comment #161)

Pushed by gmierz2@outlook.com:
https://github.com/mozilla-firefox/firefox/commit/b60c123daa3f
https://hg.mozilla.org/integration/autoland/rev/4d7843cb7023
Change browser profile location to TASK_WORKDIR in CI.
r=perftest-reviewers,aglavic

Perfherder has detected a talos performance change from push 4d7843cb7023354de9e7e1fc438b7ae220ec317d.

If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
3% tp5n nonmain_normal_fileio windows11-64-24h2-shippable e10s fission stylo webrender-sw 536,271,581.08 -> 519,213,557.42

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 47195

The following documentation link provides more information about this command.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: