Closed Bug 1772620 Opened 2 years ago Closed 11 months ago

Intermittent LeakSanitizer | leak at already_AddRefed, mozilla::dom::RemoteWorkerChild::ShutdownOnWorker, operator, decltype

Categories

(Core :: DOM: Workers, defect)

defect

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox-esr91 --- unaffected
firefox101 --- unaffected
firefox102 --- unaffected
firefox103 --- affected

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [domsecurity-intermittent][retriggered][stockwell unknown][stockwell unknown])

Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=380183940&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fJlIybxkQ4uOL3AHVg7pLQ/runs/0/artifacts/public/logs/live_backing.log


[task 2022-06-03T19:12:49.875Z] 19:12:49     INFO - TEST-START | /content-security-policy/inside-worker/sharedworker-script-src.sub.html
[task 2022-06-03T19:12:49.879Z] 19:12:49     INFO - Closing window b7d737af-50b4-43ff-90f5-f5f60be892ff
[task 2022-06-03T19:12:51.689Z] 19:12:51     INFO - ............
[task 2022-06-03T19:12:51.689Z] 19:12:51     INFO - TEST-OK | /content-security-policy/inside-worker/sharedworker-script-src.sub.html | took 1812ms
[task 2022-06-03T19:12:51.689Z] 19:12:51     INFO - Restarting browser for new test group
[task 2022-06-03T19:12:51.691Z] 19:12:51     INFO - PID 1922 | 1654283571687	Marionette	INFO	Stopped listening on port 35558
[task 2022-06-03T19:12:54.689Z] 19:12:54     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:54.690Z] 19:12:54     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:54.691Z] 19:12:54     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:54.692Z] 19:12:54     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:54.693Z] 19:12:54     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:54.693Z] 19:12:54     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:54.779Z] 19:12:54     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:54.781Z] 19:12:54     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:54.781Z] 19:12:54     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:54.782Z] 19:12:54     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:54.783Z] 19:12:54     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:54.784Z] 19:12:54     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.045Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.046Z] 19:12:55     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:55.047Z] 19:12:55     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:55.048Z] 19:12:55     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:55.050Z] 19:12:55     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:55.050Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.108Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.110Z] 19:12:55     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:55.111Z] 19:12:55     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:55.111Z] 19:12:55     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:55.111Z] 19:12:55     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:55.111Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:55.145Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:55.192Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.198Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:55.200Z] 19:12:55     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:55.200Z] 19:12:55     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:55.200Z] 19:12:55     INFO - PID 1922 |      12        384 nsComponentManagerImpl
[task 2022-06-03T19:12:55.201Z] 19:12:55     INFO - PID 1922 |       2        288 libfontconfig.so
[task 2022-06-03T19:12:55.201Z] 19:12:55     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:56.415Z] 19:12:56     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:56.416Z] 19:12:56     INFO - PID 1922 | Suppressions used:
[task 2022-06-03T19:12:56.416Z] 19:12:56     INFO - PID 1922 |   count      bytes template
[task 2022-06-03T19:12:56.417Z] 19:12:56     INFO - PID 1922 |       9        280 nsComponentManagerImpl
[task 2022-06-03T19:12:56.417Z] 19:12:56     INFO - PID 1922 |     633      18115 libfontconfig.so[task 2022-06-03T19:12:56.418Z] 19:12:56     INFO - PID 1922 |      31        189 libglib-2.0.so
[task 2022-06-03T19:12:56.418Z] 19:12:56     INFO - PID 1922 |       2        416 mozJSComponentLoader
[task 2022-06-03T19:12:56.419Z] 19:12:56     INFO - PID 1922 |       1         32 XPCWrappedNativeJSOps
[task 2022-06-03T19:12:56.419Z] 19:12:56     INFO - PID 1922 | -----------------------------------------------------
[task 2022-06-03T19:12:56.496Z] 19:12:56     INFO - Browser exited with return code 0
[task 2022-06-03T19:12:56.497Z] 19:12:56     INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 328 byte(s) leaked in 4 allocation(s).
[task 2022-06-03T19:12:56.498Z] 19:12:56     INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2022-06-03T19:12:56.498Z] 19:12:56     INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2022-06-03T19:12:56.498Z] 19:12:56     INFO - Allowed depth was 4
[task 2022-06-03T19:12:56.499Z] 19:12:56     INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived, mozilla::ipc::MessageChannel::DispatchAsyncMessage, mozilla::ipc::MessageChannel::DispatchMessage
[task 2022-06-03T19:12:56.499Z] 19:12:56     INFO - INFO | LeakSanitizer | Frame mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild matched a expected leak
[task 2022-06-03T19:12:56.499Z] 19:12:56     INFO - TEST-FAIL | LeakSanitizer | leak at detail::ProxyRelease, ProxyRelease, NS_ProxyRelease, operator
[task 2022-06-03T19:12:56.499Z] 19:12:56     INFO - INFO | LeakSanitizer | Frame detail::ProxyRelease matched a expected leak
[task 2022-06-03T19:12:56.500Z] 19:12:56     INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at already_AddRefed, mozilla::dom::RemoteWorkerChild::ShutdownOnWorker, operator, decltype
[task 2022-06-03T19:12:56.501Z] 19:12:56     INFO - TEST-FAIL | LeakSanitizer | leak at mozilla::SupportsThreadSafeWeakPtr, mozilla::dom::RemoteWorkerChild::RemoteWorkerChild, mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived
[task 2022-06-03T19:12:56.501Z] 19:12:56     INFO - INFO | LeakSanitizer | Frame mozilla::SupportsThreadSafeWeakPtr matched a expected leak
[task 2022-06-03T19:12:56.503Z] 19:12:56     INFO - Closing logging queue
[task 2022-06-03T19:12:56.503Z] 19:12:56     INFO - queue closed
[task 2022-06-03T19:12:56.516Z] 19:12:56     INFO - INFO | runtests.py | ASan using symbolizer at /builds/worker/workspace/build/application/firefox/llvm-symbolizer
[task 2022-06-03T19:12:56.521Z] 19:12:56     INFO - LSan enabled.
[task 2022-06-03T19:12:56.522Z] 19:12:56     INFO - INFO | runtests.py | ASan running in default memory configuration
[task 2022-06-03T19:12:56.522Z] 19:12:56     INFO - UBSan enabled.
[task 2022-06-03T19:12:56.522Z] 19:12:56     INFO - Application command: /builds/worker/workspace/build/application/firefox/firefox --marionette about:blank -profile /tmp/tmpjxhbjasa
[task 2022-06-03T19:12:56.528Z] 19:12:56     INFO - PID 2225 | 1654283552572	Marionette	INFO	Marionette enabled
[task 2022-06-03T19:12:56.529Z] 19:12:56     INFO - PID 2225 | 1654283552580	Marionette	INFO	Listening on port 49588
[task 2022-06-03T19:12:56.529Z] 19:12:56     INFO - PID 2225 | console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/tmpf7cll0pk/search.json.mozlz4", (void 0)))
[task 2022-06-03T19:12:56.530Z] 19:12:56     INFO - Starting runner
[task 2022-06-03T19:12:57.486Z] 19:12:57     INFO - TEST-START | /content-security-policy/unsafe-eval/eval-allowed.sub.html

Based on the above retriggers and the changes in Bug 1743020 looks like the leaks in Bug 1772309 turned into these ones. Looks like they leak on content-security-policy/inside-worker/sharedworker-script-src.sub.html
Nika, could you please take a look over this?

Flags: needinfo?(nika)
Regressed by: 1743020

Set release status flags based on info from the regressing bug 1743020

Has Regression Range: --- → yes

There have been 55 failures within the last 7 days, all of them on Linux 18.04 x64 WebRender asan opt.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=381055856&repo=autoland&lineNumber=2944

[task 2022-06-12T21:24:24.010Z] 21:24:24    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived, mozilla::ipc::MessageChannel::DispatchAsyncMessage, mozilla::ipc::MessageChannel::DispatchMessage
[task 2022-06-12T21:24:24.010Z] 21:24:24    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at detail::ProxyRelease, ProxyRelease, NS_ProxyRelease, operator
[task 2022-06-12T21:24:24.010Z] 21:24:24    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at already_AddRefed, mozilla::dom::RemoteWorkerChild::ShutdownOnWorker, operator, decltype
[task 2022-06-12T21:24:24.011Z] 21:24:24    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::SupportsThreadSafeWeakPtr, mozilla::dom::RemoteWorkerChild::RemoteWorkerChild, mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived
Whiteboard: [domsecurity-intermittent][retriggered] → [domsecurity-intermittent][retriggered][stockwell needswork]

(In reply to Cosmin Sabou [:CosminS] from comment #4)

Based on the above retriggers and the changes in Bug 1743020 looks like the leaks in Bug 1772309 turned into these ones. Looks like they leak on content-security-policy/inside-worker/sharedworker-script-src.sub.html
Nika, could you please take a look over this?

Definitely sounds like these are existing leaks which just changed in signature due to the new library being used.

These leaks appear to be related to worker lifetimes around shutdown or similar, so redirecting over to that team to take a look at them.

Has Regression Range: yes → ---
Component: DOM: Security → DOM: Workers
Flags: needinfo?(nika) → needinfo?(bugmail)
Keywords: regression
No longer regressed by: 1743020
See Also: → 1743020

There have been 41 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

Update:

There have been 35 failures within the last 7 days on Linux 18.04 x64 WebRender asan opt.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=384579280&repo=mozilla-central&lineNumber=6089

[task 2022-07-17T10:00:33.010Z] 10:00:33     INFO - TEST-INFO | Main app process: exit 0
[task 2022-07-17T10:00:33.010Z] 10:00:33     INFO - TEST-INFO | LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2022-07-17T10:00:33.010Z] 10:00:33     INFO - TEST-INFO | LeakSanitizer | This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2022-07-17T10:00:33.010Z] 10:00:33    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at already_AddRefed, mozilla::dom::RemoteWorkerChild::ShutdownOnWorker, operator, decltype
[task 2022-07-17T10:00:33.010Z] 10:00:33    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::SupportsThreadSafeWeakPtr, mozilla::dom::RemoteWorkerChild::RemoteWorkerChild, mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived
[task 2022-07-17T10:00:33.011Z] 10:00:33    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at detail::ProxyRelease, ProxyRelease, NS_ProxyRelease, operator
[task 2022-07-17T10:00:33.012Z] 10:00:33    ERROR - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::ipc::BackgroundChildImpl::AllocPRemoteWorkerChild, mozilla::ipc::PBackgroundChild::OnMessageReceived, mozilla::ipc::MessageChannel::DispatchAsyncMessage, mozilla::ipc::MessageChannel::DispatchMessage
[task 2022-07-17T10:00:33.013Z] 10:00:33     INFO - runtests.py | Application ran for: 0:02:38.659438

There have been 34 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

There have been 35 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

There have been 42 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

There have been 33 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

There have been 42 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-asan-qr

This family of leaks was probably related to bug 1775784, but since the failures went away well before that bug was fixed, it's likely either:

  • This signature changed to one of the signatures that were much more egregious but related to bug 1775784.
  • This signature was related to changed in the worker script loader. A number of changes landed through 2022-11 that could have aided in the signature disappearing (or shifting).
Status: NEW → RESOLVED
Closed: 11 months ago
Flags: needinfo?(bugmail)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.