1856091 - Intermittent SUMMARY: ThreadSanitizer: data race /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:350:36 in operator!

Status: RESOLVED FIXED

(Core :: DOM: File, defect)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=430834030&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aOn8obgdQuCSs362LDehEA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aOn8obgdQuCSs362LDehEA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2023-09-29T22:59:04.600Z] 22:59:04     INFO -   Thread T17 'StreamTrans #1' (tid=1822, running) created by main thread at:
[task 2023-09-29T22:59:04.601Z] 22:59:04     INFO -     #0 pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1020:3 (firefox-bin+0xc558b) (BuildId: b7378e9b5df13b13e9b3b1bb48cf4e2bf7d8a9c9)
[task 2023-09-29T22:59:04.602Z] 22:59:04     INFO -     #1 _PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:458:14 (libnspr4.so+0x4242e) (BuildId: b6f58db5ff0819afc822840d196a0175d36eee04)
[task 2023-09-29T22:59:04.603Z] 22:59:04     INFO -     #2 PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:533:12 (libnspr4.so+0x376c4) (BuildId: b6f58db5ff0819afc822840d196a0175d36eee04)
[task 2023-09-29T22:59:04.604Z] 22:59:04     INFO -     #3 nsThread::Init(nsTSubstring<char> const&) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:619:20 (libxul.so+0x33a1527) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.605Z] 22:59:04     INFO -     #4 nsThreadManager::NewNamedThread(nsTSubstring<char> const&, nsIThreadManager::ThreadCreationOptions, nsIThread**) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadManager.cpp:597:22 (libxul.so+0x33a9596) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.606Z] 22:59:04     INFO -     #5 NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, already_AddRefed<nsIRunnable>, nsIThreadManager::ThreadCreationOptions) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:176:57 (libxul.so+0x33b214f) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.608Z] 22:59:04     INFO -     #6 NS_NewNamedThread /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:168:10 (libxul.so+0x33ac55f) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.609Z] 22:59:04     INFO -     #7 nsThreadPool::PutEvent(already_AddRefed<nsIRunnable>, unsigned int) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp:126:17 (libxul.so+0x33ac55f)
[task 2023-09-29T22:59:04.610Z] 22:59:04     INFO -     #8 Dispatch /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp:380:3 (libxul.so+0x33adec4) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.611Z] 22:59:04     INFO -     #9 non-virtual thunk to nsThreadPool::Dispatch(already_AddRefed<nsIRunnable>, unsigned int) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp (libxul.so+0x33adec4)
[task 2023-09-29T22:59:04.612Z] 22:59:04     INFO -     #10 Dispatch /builds/worker/checkouts/gecko/netwerk/base/nsStreamTransportService.cpp:296:16 (libxul.so+0x35f2fa9) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.613Z] 22:59:04     INFO -     #11 non-virtual thunk to mozilla::net::nsStreamTransportService::Dispatch(already_AddRefed<nsIRunnable>, unsigned int) /builds/worker/checkouts/gecko/netwerk/base/nsStreamTransportService.cpp (libxul.so+0x35f2fa9)
[task 2023-09-29T22:59:04.614Z] 22:59:04     INFO -     #12 mozilla::MemoryTelemetry::GatherReports(std::function<void ()> const&) /builds/worker/checkouts/gecko/xpcom/base/MemoryTelemetry.cpp:304:30 (libxul.so+0x32abb76) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.615Z] 22:59:04     INFO -     #13 mozilla::MemoryTelemetry::DelayedInit() /builds/worker/checkouts/gecko/xpcom/base/MemoryTelemetry.cpp:126:3 (libxul.so+0x32ab53c) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.616Z] 22:59:04     INFO -     #14 (anonymous namespace)::TelemetryImpl::DelayedInit() /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/Telemetry.cpp:1663:26 (libxul.so+0x9df71a7) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.617Z] 22:59:04     INFO -     #15 NS_InvokeByIndex /builds/worker/checkouts/gecko/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:101 (libxul.so+0x33d1645) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.618Z] 22:59:04     INFO -     #16 XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) /builds/worker/checkouts/gecko/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:966:10 (libxul.so+0x4020b7c) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.619Z] 22:59:04     INFO -     #17 CallJSNative /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:486:13 (libxul.so+0xa0b8599) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.620Z] 22:59:04     INFO -     #18 js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:580:12 (libxul.so+0xa0b8599)
[task 2023-09-29T22:59:04.621Z] 22:59:04     INFO -     #19 InternalCall /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:647:10 (libxul.so+0xa0c8e2b) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.622Z] 22:59:04     INFO -     #20 CallFromStack /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:652:10 (libxul.so+0xa0c8e2b)
[task 2023-09-29T22:59:04.623Z] 22:59:04     INFO -     #21 js::Interpret(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:3396:16 (libxul.so+0xa0c8e2b)
[task 2023-09-29T22:59:04.623Z] 22:59:04     INFO -     #22 MaybeEnterInterpreterTrampoline /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:400:10 (libxul.so+0xa0b7c0c) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.624Z] 22:59:04     INFO -     #23 js::RunScript(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:458:13 (libxul.so+0xa0b7c0c)
[task 2023-09-29T22:59:04.625Z] 22:59:04     INFO -     #24 js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:612:13 (libxul.so+0xa0b8666) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.626Z] 22:59:04     INFO -     #25 InternalCall /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:647:10 (libxul.so+0xa0b9217) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.627Z] 22:59:04     INFO -     #26 js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:679:8 (libxul.so+0xa0b9217)
[task 2023-09-29T22:59:04.628Z] 22:59:04     INFO -     #27 JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) /builds/worker/checkouts/gecko/js/src/vm/CallAndConstruct.cpp:119:10 (libxul.so+0xa15ae29) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.629Z] 22:59:04     INFO -     #28 mozilla::dom::IdleRequestCallback::Call(mozilla::dom::BindingCallContext&, JS::Handle<JS::Value>, mozilla::dom::IdleDeadline&, mozilla::ErrorResult&) /builds/worker/workspace/obj-build/dom/bindings/./WindowBinding.cpp:453:8 (libxul.so+0x564482f) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.630Z] 22:59:04     INFO -     #29 Call /builds/worker/workspace/obj-build/dist/include/mozilla/dom/WindowBinding.h:397:12 (libxul.so+0x4acf8da) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.631Z] 22:59:04     INFO -     #30 Call /builds/worker/workspace/obj-build/dist/include/mozilla/dom/WindowBinding.h:410:12 (libxul.so+0x4acf8da)
[task 2023-09-29T22:59:04.632Z] 22:59:04     INFO -     #31 mozilla::dom::(anonymous namespace)::IdleDispatchRunnable::Run() /builds/worker/checkouts/gecko/dom/base/ChromeUtils.cpp:482:17 (libxul.so+0x4acf8da)
[task 2023-09-29T22:59:04.633Z] 22:59:04     INFO -     #32 mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:559:16 (libxul.so+0x338b892) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.634Z] 22:59:04     INFO -     #33 mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:886:26 (libxul.so+0x33821c0) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.635Z] 22:59:04     INFO -     #34 mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:751:15 (libxul.so+0x3380af8) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.636Z] 22:59:04     INFO -     #35 mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:495:36 (libxul.so+0x3380cbf) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.637Z] 22:59:04     INFO -     #36 operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:218:37 (libxul.so+0x338e744) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.638Z] 22:59:04     INFO -     #37 mozilla::detail::RunnableFunction<mozilla::TaskController::TaskController()::$_0>::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.h:548:5 (libxul.so+0x338e744)
[task 2023-09-29T22:59:04.639Z] 22:59:04     INFO -     #38 nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1198:16 (libxul.so+0x33a42ca) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.640Z] 22:59:04     INFO -     #39 NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:480:10 (libxul.so+0x33aaa64) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.641Z] 22:59:04     INFO -     #40 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21 (libxul.so+0x3ea290e) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.642Z] 22:59:04     INFO -     #41 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:268:30 (libxul.so+0x3ea33db) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.643Z] 22:59:04     INFO -     #42 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:370:10 (libxul.so+0x3e1ce98) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.644Z] 22:59:04     INFO -     #43 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:363:3 (libxul.so+0x3e1ce98)
[task 2023-09-29T22:59:04.645Z] 22:59:04     INFO -     #44 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:345:3 (libxul.so+0x3e1ce98)
[task 2023-09-29T22:59:04.646Z] 22:59:04     INFO -     #45 nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:148:27 (libxul.so+0x7c72e53) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.647Z] 22:59:04     INFO -     #46 XRE_RunAppShell() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:721:20 (libxul.so+0x9f3ed8f) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.648Z] 22:59:04     INFO -     #47 mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:235:9 (libxul.so+0x3ea338a) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.649Z] 22:59:04     INFO -     #48 RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:370:10 (libxul.so+0x3e1ce98) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.649Z] 22:59:04     INFO -     #49 RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:363:3 (libxul.so+0x3e1ce98)
[task 2023-09-29T22:59:04.650Z] 22:59:04     INFO -     #50 MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:345:3 (libxul.so+0x3e1ce98)
[task 2023-09-29T22:59:04.651Z] 22:59:04     INFO -     #51 XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:656:34 (libxul.so+0x9f3ea9d) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.652Z] 22:59:04     INFO -     #52 mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/Bootstrap.cpp:67:12 (libxul.so+0x9f4abd2) (BuildId: a0675658408969349d52fa1c85c9ace1beebdcae)
[task 2023-09-29T22:59:04.653Z] 22:59:04     INFO -     #53 content_process_main /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28 (firefox-bin+0x14e442) (BuildId: b7378e9b5df13b13e9b3b1bb48cf4e2bf7d8a9c9)
[task 2023-09-29T22:59:04.654Z] 22:59:04     INFO -     #54 main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:375:18 (firefox-bin+0x14e442)
[task 2023-09-29T22:59:04.655Z] 22:59:04     INFO - SUMMARY: ThreadSanitizer: data race /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:350:36 in operator!
[task 2023-09-29T22:59:04.656Z] 22:59:04     INFO - ==================
[task 2023-09-29T22:59:04.664Z] 22:59:04    ERROR - A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
[task 2023-09-29T22:59:08.116Z] 22:59:08     INFO - 1696028348114	Marionette	TRACE	Received observer notification quit-application
[task 2023-09-29T22:59:08.117Z] 22:59:08     INFO - 1696028348115	Marionette	INFO	Stopped listening on port 2828
[task 2023-09-29T22:59:08.122Z] 22:59:08     INFO - 1696028348121	Marionette	DEBUG	Marionette stopped listening
[task 2023-09-29T22:59:08.413Z] 22:59:08     INFO - [Parent 1517, IPC I/O Parent] WARNING: process 1802 exited on signal 6: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_posix.cc:265
[task 2023-09-29T22:59:08.524Z] 22:59:08     INFO - !!! error running onStopped callback: TypeError: callback is not a function
[task 2023-09-29T22:59:08.626Z] 22:59:08     INFO - console.error: ({})
[task 2023-09-29T22:59:12.016Z] 22:59:12     INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/fs/test/crashtests/1841702.html (finished) | application terminated with exit code 245
[task 2023-09-29T22:59:12.016Z] 22:59:12     INFO -  mozlog.structuredlog: Failure calling log handler:
[task 2023-09-29T22:59:12.016Z] 22:59:12     INFO -  Traceback (most recent call last):
[task 2023-09-29T22:59:12.017Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/structuredlog.py", line 318, in _handle_log
[task 2023-09-29T22:59:12.017Z] 22:59:12     INFO -      handler(data)
[task 2023-09-29T22:59:12.017Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/handlers/base.py", line 74, in __call__
[task 2023-09-29T22:59:12.018Z] 22:59:12     INFO -      formatted = self.formatter(data)
[task 2023-09-29T22:59:12.018Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/handlers/base.py", line 42, in __call__
[task 2023-09-29T22:59:12.019Z] 22:59:12     INFO -      return self.inner(item)
[task 2023-09-29T22:59:12.019Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/formatters/errorsummary.py", line 30, in __call__
[task 2023-09-29T22:59:12.020Z] 22:59:12     INFO -      rv = BaseFormatter.__call__(self, data)
[task 2023-09-29T22:59:12.020Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/reader.py", line 72, in __call__
[task 2023-09-29T22:59:12.021Z] 22:59:12     INFO -      return handler(data)
[task 2023-09-29T22:59:12.021Z] 22:59:12     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.7/site-packages/mozlog/formatters/errorsummary.py", line 134, in shutdown_failure
[task 2023-09-29T22:59:12.022Z] 22:59:12     INFO -      data["group"] = [g for g in self.groups if item["group"].endswith(g)][0]
[task 2023-09-29T22:59:12.022Z] 22:59:12     INFO -  IndexError: list index out of range
[task 2023-09-29T22:59:12.030Z] 22:59:12     INFO - REFTEST INFO | Process mode: e10s
[task 2023-09-29T22:59:12.055Z] 22:59:12     INFO - Return code: 245
[task 2023-09-29T22:59:12.057Z] 22:59:12    ERROR - No suite end message was emitted by this harness.
[task 2023-09-29T22:59:12.057Z] 22:59:12     INFO - TinderboxPrint: reftest-crashtest<br/>217/0/0
[task 2023-09-29T22:59:12.057Z] 22:59:12  WARNING - setting return code to 2

Comment 1

[Andrew McCreight [:mccr8]]

This looks like a race on RemoteQuotaObject::mActor.

Comment 2

[Andrew McCreight [:mccr8]]

Attachment: full TSan report

Comment 3

[Andrew McCreight [:mccr8]]

I'll mark this moderate under the assumption that it is hard to exploit.

Comment 4

[Kagami Rosylight [:saschanaz] (they/them)]

I guess this is also essentially DOM: File?

Comment 5

[Jan Varga [:janv]]

I'm simplifying this code in bug 1857155. Some of the reported problems should be fixed by that.

Comment 6

[Jan Varga [:janv]]

No recent failures.

Comment 7

[Daniel Veditz [:dveditz]]

Bulk-unhiding security bugs fixed in Firefox 119-121 (Fall 2023). Use "moo-doctrine-subsidy" to filter