Very frequent crashes when the PWAsForFirefox extension is enabled [extension native messaging]
Categories
(Toolkit :: Async Tooling, defect, P1)
Tracking
()
People
(Reporter: filip.stamcar, Assigned: robwu)
References
(Blocks 1 open bug, Regression)
Details
(4 keywords, Whiteboard: [addons-jira][adv-main144+][adv-esr140.4-])
Crash Data
Attachments
(2 files, 2 external files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0
Steps to reproduce:
- Install and configure the PWAsForFirefox extension on Windows: https://addons.mozilla.org/firefox/addon/pwas-for-firefox/
- Browse like normal: Open tabs, click links, play videos, etc.
- At some point the browser just crashes
Actual results:
The browser crashes. The crashes happen seemingly at random, while clicking on links or opening new websites, even while just waiting. Sometimes, multiple crashes happen just in a few minutes, but sometimes, it works normal for quite some time before it crashes.
I'm using Nightly 145.0a1. The crashes started happening for me on 2025-09-28 (or maybe one build before that) and are still happening on 2025-09-30 (when the extension is enabled). These are my recent crash reports:
- bp-30314d88-2ca2-4074-b7e1-1c1250251001 1. 10. 2025, 17:20
- bp-a5d24abd-7b3a-4d9c-b16e-0cb340251001 1. 10. 2025, 17:18
- bp-4491b383-98bb-44ce-92ea-618dd0251001 1. 10. 2025, 17:16
- bp-caf3f5e3-674c-437f-b831-ed5380250930 30. 09. 2025, 19:02
- bp-d4cfa56b-095e-4e67-9786-752cc0250930 30. 09. 2025, 18:57
- bp-3d0afebd-672b-4ad1-a943-b57a40250930 30. 09. 2025, 18:51
- bp-c44bf12a-96d1-4cf7-b97b-1533b0250930 30. 09. 2025, 17:10
- bp-6e4e723a-ee44-4c31-9c8f-218560250930 30. 09. 2025, 14:15
- bp-06e66a30-78c2-4c92-aa6b-716320250930 30. 09. 2025, 13:50
- bp-19b8c291-d4fc-418d-817e-288000250930 30. 09. 2025, 13:46
- bp-f41921ba-d3a0-43c2-8a56-c8f650250930 30. 09. 2025, 13:41
- bp-80f52ece-ac55-47d3-9ad0-411630250930 30. 09. 2025, 13:35
- bp-e03f3d5b-9ca5-4cba-94e6-9094b0250930 30. 09. 2025, 13:21
- bp-5faa043c-fda3-404c-92d3-d00a00250930 30. 09. 2025, 12:47
- bp-cbc259d6-ac52-4bb5-8d0b-0a50b0250930 30. 09. 2025, 12:42
- bp-b8b443bf-46b1-4a89-9c11-324b70250930 30. 09. 2025, 12:32
- bp-06fe344e-93a9-44cd-92f2-d948e0250930 30. 09. 2025, 12:31
- bp-b4e3bce3-bc6b-430b-ab14-f2a500250930 30. 09. 2025, 12:30
- bp-9d6cc039-6021-47b7-be5a-29b390250930 30. 09. 2025, 12:29
- bp-92831fa8-ef20-40aa-ad31-8e53e0250929 29. 09. 2025, 21:31
- bp-68847399-e8a1-455a-aefc-895ee0250929 29. 09. 2025, 20:40
- bp-df7888b0-6d51-4ba1-ae57-e19b90250929 29. 09. 2025, 20:33
- bp-81b17e4d-eff2-4efa-afc6-4f50f0250929 29. 09. 2025, 19:44
- bp-aa744619-dd32-4359-b99d-db81a0250929 29. 09. 2025, 19:36
- bp-b07775c7-658b-4618-9aec-446760250929 29. 09. 2025, 18:42
- bp-95553eec-abad-47fe-8c9c-8bc200250929 29. 09. 2025, 18:40
- bp-887b586a-5393-4830-94b4-983cd0250929 29. 09. 2025, 17:38
- bp-1646e224-4f33-4642-abca-d32fc0250929 29. 09. 2025, 17:37
- bp-aef68f27-f031-42c8-9900-f184d0250929 29. 09. 2025, 17:28
- bp-6acfe661-27d6-4007-acd0-412740250929 29. 09. 2025, 17:26
- bp-505d758d-ed10-4c27-ab82-cac720250929 29. 09. 2025, 17:10
- bp-149048bb-eb62-4467-9b22-a4e7b0250929 29. 09. 2025, 16:02
- bp-43dcd227-ca3a-4b74-b190-e70df0250929 29. 09. 2025, 16:00
- bp-6c28c20c-fe29-4847-94e1-12d110250929 29. 09. 2025, 15:55
- bp-4b715cbf-2777-497c-be18-97d3f0250929 29. 09. 2025, 15:45
- bp-9ce9f2c7-ec03-45a3-a63e-bbdf40250929 29. 09. 2025, 15:34
- bp-037d40c9-f34f-4545-a2f3-c52b00250929 29. 09. 2025, 15:17
- bp-dc72e270-ce81-4941-95d3-0501c0250929 29. 09. 2025, 15:16
- bp-f02dba17-ebc5-4dd2-b42c-4bc7e0250929 29. 09. 2025, 15:05
- bp-0ee81203-a558-4dd4-aeaa-69a1a0250929 29. 09. 2025, 14:34
- bp-3b328421-fa02-4aa9-99d5-f4a7b0250929 29. 09. 2025, 14:21
- bp-f6bc7909-0474-4003-8e33-34fd60250929 29. 09. 2025, 14:21
- bp-d52dd684-481e-45f8-855b-4d9510250929 29. 09. 2025, 13:48
- bp-4be07dcd-82a8-4cf0-afa6-f34980250929 29. 09. 2025, 13:31
- bp-1f029316-d950-4060-adf6-f5e9f0250929 29. 09. 2025, 13:27
- bp-cfd94c6a-ed22-42b8-ad65-37b610250929 29. 09. 2025, 13:27
- bp-63df2ebd-7761-48ec-9220-220870250929 29. 09. 2025, 13:26
- bp-9a789a4d-94c3-4b55-bc8c-c69940250929 29. 09. 2025, 13:08
- bp-ee252519-bd9e-4589-9dfd-d6ec40250929 29. 09. 2025, 12:30
- bp-695772d3-40d4-4cdb-be04-a11fd0250929 29. 09. 2025, 12:13
- bp-99e3529f-a5b1-40be-a229-6fece0250929 29. 09. 2025, 11:40
- bp-e2baf0b4-a313-453f-a6c8-d0f230250929 29. 09. 2025, 11:34
- bp-9bfe862f-eae4-42cc-b5f0-65cf00250929 29. 09. 2025, 11:33
- bp-94f963f9-1dce-42b0-ac94-748b40250929 29. 09. 2025, 11:27
- bp-d4583a03-c176-478d-932f-44d5b0250929 29. 09. 2025, 11:26
- bp-1d15d076-1ec2-4b67-83f5-942b70250929 29. 09. 2025, 11:24
- bp-74ebf94b-df3c-400c-8ffe-ac8b30250929 29. 09. 2025, 11:15
- bp-6a5d127d-ee87-4c8c-a74d-bbecd0250929 29. 09. 2025, 10:31
- bp-0b35fd83-d0b1-48b7-9ff2-990df0250929 29. 09. 2025, 10:31
- bp-6cf2bf59-dffc-4c49-ace5-314a20250928 28. 09. 2025, 22:09
- bp-0a5787e2-729f-492b-a390-4704c0250928 28. 09. 2025, 22:08
- bp-7abd7647-3bfd-4e84-99b2-9420a0250928 28. 09. 2025, 22:00
- bp-ea847f8b-a33e-4dbc-9d0b-84dd60250928 28. 09. 2025, 21:54
- bp-58657e17-f156-4c28-9e13-6aa120250928 28. 09. 2025, 21:50
- bp-25c21016-ef96-468f-ad99-037d30250928 28. 09. 2025, 21:48
- bp-009f93c8-28b9-4ddb-b385-4a20c0250928 28. 09. 2025, 21:35
- bp-acdbdaa9-771d-4f7d-90b9-eb0020250928 28. 09. 2025, 21:33
Crashes have different signatures, but the common ones seem to be:
- [@ js::gc::detail::ChunkPtrHasStoreBuffer ]
- [@ mozilla::detail::EntrySlot<T>::isFree ]
- [@ js::jit::CacheIRStubInfo::fieldType ]
I've initially reported the crash in this comment.
One other user of the extension reported this issue on GitHub. However, for them, the problems started with the update to Firefox 143. These are some of their crash reports:
- bp-6813e25a-7f01-4b4d-b72d-ad9290251001 (FF 143.0.1)
- bp-b5bcde4a-b412-4f54-a247-514230251001 (FF 143.0.3)
It appears this happens only on Windows. It happens with PWAsForFirefox 2.16.0 (which was released last week), 2.15.0, and also when using my development build. When the extension is disabled, crashes do not occur.
Expected results:
Firefox shouldn't crash.
| Reporter | ||
Updated•1 year ago
|
Comment 1•1 year ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.
| Reporter | ||
Comment 2•1 year ago
|
||
I don't think it's related to video playback. It happens even when no video/audio is playing.
Comment 3•1 year ago
•
|
||
So this extension is causing Firefox to crash frequently: https://github.com/filips123/PWAsForFirefox
Rob, do you have thoughts on this? We don't expose any low-level APIs to extension code right?
For an additional data point, I have this addon installed on my linux laptop but I don't have similar crashes.
Comment 5•1 year ago
|
||
I haven't been able to reproduce this yet on my Windows machine.
Filip, do you see this with a particular website or any website? Should I change any settings of the add-on after installing it?
| Reporter | ||
Comment 6•1 year ago
|
||
It happens regardless of website. When the extension setup wizard is open (in my case, moz-extension://657e0440-4d7d-4abb-add9-32789d0686ce/setup/install.html, not sure if the extension ID is always the same), it seems to always crash in a few minutes. But even when no extension-related pages/popups are open and I'm just browsing websites normally, it also frequently crashes.
The native messaging host part should be installed. Other settings can be default.
| Reporter | ||
Comment 7•1 year ago
|
||
Forgot to add, crashes also happens in a fresh profile of the same Firefox Nightly installation with PWAsForFirefox installed after a few minutes of browsing.
- bp-8c2b225a-09f6-472d-8646-0fdf70251003 3. 10. 2025, 16:10
- bp-f6ea3665-2da8-4100-872d-db1a60251003 3. 10. 2025, 16:10
- bp-b4050b5c-d4cf-469a-85ff-2e21f0251003 3. 10. 2025, 16:13
Comment 8•1 year ago
|
||
I have version 2.16.0 of PWAsForFirefox installed. Is that the same version you have?
Would you be able to try this with a Firefox ASan build? It might give us more information about the crash. Here is a recent Nightly ASan build.
Another option is a Windows debug build. Run firefox.exe from the command line and see if it prints any assertion failures or other errors when it crashes.
| Reporter | ||
Comment 9•1 year ago
|
||
Yes, I'm using PWAsForFirefox 2.16.0.
With the ASan build, the crash still didn't happen after around 10 min of browsing. I don't know if this is related, but when I open the extension popup, I see messages like this in the command line:
Worker debug: IOCP notification for unknown pipe: 356
Worker debug: IOCP notification for unknown pipe: 370
With the debug build, the crash still happened. Most recent lines from the log:
[Parent 14636, Main Thread] WARNING: '!lock.ref()->Remove(tuple)', file /builds/worker/checkouts/gecko/toolkit/components/glean/bindings/private/TimingDistribution.cpp:284
[Parent 14636, Main Thread] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x804B000A (NS_ERROR_MALFORMED_URI): file /builds/worker/checkouts/gecko/dom/base/Document.cpp:17407
[Parent 14636, Main Thread] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /builds/worker/checkouts/gecko/parser/html/nsHtml5StreamParser.cpp:1230
[Parent 14636, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:95
Worker debug: IOCP notification for unknown pipe: 2
Worker debug: IOCP notification for unknown pipe: 4
[14636] Assertion failure: chunk->runtime, at /builds/worker/workspace/obj-build/dist/include/js/HeapAPI.h:646
#01: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x84cac2b]
#02: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86515c6]
#03: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8ceb96e]
#04: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86557a6]
#05: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86553db]
#06: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85ea42b]
#07: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85ea30b]
#08: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x864b0ad]
#09: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x864ab7f]
#10: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e976c]
#11: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e92bf]
#12: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e91b7]
#13: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8632cae]
#14: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8638539]
#15: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x863c46e]
#16: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x863dda5]
#17: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2e6ea8e]
#18: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2e6ebca]
#19: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b64f18]
#20: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b645fc]
#21: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b74542]
#22: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x9f9cc0]
#23: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x9fa98f]
#24: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa0c138]
#25: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa08b92]
#26: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa08e4a]
#27: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa1ca57]
#28: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa3795c]
#29: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa3f9a1]
#30: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x16d4268]
#31: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x164591f]
#32: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x16457ef]
#33: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x6e3d8c8]
#34: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x6f6643e]
#35: GIFFT_LabeledTimingDistributionCancel[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8152877]
#36: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830a8b8]
#37: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830d753]
#38: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830e440]
#39: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0x17eb]
#40: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0x1247]
#41: TargetNtUnmapViewOfSection[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0xaf888]
#42: BaseThreadInitThunk[C:\WINDOWS\System32\KERNEL32.DLL +0x2e8d7]
| Reporter | ||
Comment 10•1 year ago
|
||
Comment 11•1 year ago
|
||
Thanks, that's very helpful.
I'm still not able to reproduce this on my Windows 11 machine. I'll try installing the latest updates...
I'm also going to hide this bug just in case this turns out to be a bad Firefox bug.
Comment 12•1 year ago
|
||
(In reply to Filip Štamcar from comment #6)
The native messaging host part should be installed. Other settings can be default.
Hey, is there some documentation about what the native part does? In particular how does it communicate with the extension?
Updated•1 year ago
|
| Reporter | ||
Comment 13•1 year ago
|
||
Hey, is there some documentation about what the native part does? In particular how does it communicate with the extension?
A high-level description is available on the docs site. It communicates with the extension using browser.runtime.sendNativeMessage. The code for the native part is here (and in other files in the connector directory). In the extension, it's called from various places, for example, here on the setup page to check some program versions.
Comment 14•1 year ago
|
||
Another thing you could try (until one of us is able to reproduce this..) is mozregression, to narrow down when this started happening on the Firefox side.
| Reporter | ||
Comment 15•1 year ago
|
||
I ran mozregression on range 2024-10-03 - 2025-10-03. I marked the build as good if it didn't crash after around 10 min of browsing. It found the following commit: Bug 1982950 - Stop polling as soon as process exits r=gstoll (https://phabricator.services.mozilla.com/D261060)
This is the progress summary:
Bisecting on mozilla-central [2024-10-03 - 2025-10-03]
Tested mozilla-central build: 2025-04-04 (verdict: g)
Tested mozilla-central build: 2025-07-04 (verdict: g)
Tested mozilla-central build: 2025-08-19 (verdict: b)
Tested mozilla-central build: 2025-07-27 (verdict: g)
Tested mozilla-central build: 2025-08-08 (verdict: g)
Tested mozilla-central build: 2025-08-14 (verdict: b)
Tested mozilla-central build: 2025-08-11 (verdict: g)
Tested mozilla-central build: 2025-08-13 (verdict: g)
Bisecting on mozilla-central [968f8460 - 7de8dcc3]
Tested mozilla-central build: 48334adb (verdict: g)
Bisecting on autoland [48334adb - 7de8dcc3]
Tested autoland build: b270d013 (verdict: g)
Tested autoland build: 172b4f01 (verdict: g)
Tested autoland build: d67fc3a9 (verdict: g)
Tested autoland build: 1387ca28 (verdict: b)
Tested autoland build: 1fad8465 (verdict: g)
I don't know if this is relevant, but on builds b270d0133ee593f14f63d4515b602fba893b9dbc and 172b4f01ef69ad5af1fc8bc8dd2f42193775a4b8, native messaging seems to be broken. Any extension call to browser.runtime.sendNativeMessage failed with the following error:
null ExtensionCommon.sys.mjs:811
normalizeError resource://gre/modules/ExtensionCommon.sys.mjs:811
wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:964
(Async: promise callback)
wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:921
wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:920
callAsyncFunction resource://gre/modules/ExtensionCommon.sys.mjs:1159
callAsyncFunction resource://gre/modules/ExtensionChild.sys.mjs:768
callAndLog resource://gre/modules/ExtensionChild.sys.mjs:748
callAsyncFunction resource://gre/modules/ExtensionChild.sys.mjs:767
stub resource://gre/modules/Schemas.sys.mjs:3048
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:714
(Async: setTimeout handler)
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
AsyncFunctionThrow self-hosted:804
(Async: async)
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
AsyncFunctionThrow self-hosted:804
(Async: async)
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
AsyncFunctionThrow self-hosted:804
(Async: async)
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
AsyncFunctionThrow self-hosted:804
(Async: async)
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
AsyncFunctionThrow self-hosted:804
(Async: async)
checkLicenseAgreement moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:682
InterpretGeneratorResume self-hosted:1332
AsyncFunctionNext self-hosted:800
(Async: async)
startWizard moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:795
InterpretGeneratorResume self-hosted:1332
AsyncFunctionNext self-hosted:800
(Async: async)
lMWoA moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:797
newRequire moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:84
<anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:144
<anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:163
Error: An unexpected error occurred
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:714
setTimeout handler*checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
checkLicenseAgreement moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:682
startWizard moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:795
lMWoA moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:797
newRequire moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:84
<anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:144
<anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:163
install.c58c8571.js:742:21
I marked these two builds as good as the browser didn't crash. In all other builds, native messaging worked fine (apart from the crash).
It's possible that some of the good builds could still crash after more time. However, on every bad build, the crash happened in less than 5 min and I waited around 10 min, so I think this is unlikely. And the result seems consistent with the bug report of the other extension user that said the crashes started with Firefox 143. However, it's weird that I was using Nightly for the whole time, but the crashes only started for me last week. I have crash report bp-26a9aca7-af15-4554-92b0-8d7bb0250816 from around the time the patch was introduced (2025-08-16) and it has the same signature as some of the newer crashes ([@ js::gc::detail::ChunkPtrHasStoreBuffer ]), but that was the last crash until now, even though I had the extension enabled and also used it.
Updated•1 year ago
|
Comment 16•1 year ago
|
||
(In reply to Filip Štamcar from comment #15)
I ran mozregression on range 2024-10-03 - 2025-10-03. I marked the build as good if it didn't crash after around 10 min of browsing. It found the following commit: Bug 1982950 - Stop polling as soon as process exits r=gstoll (https://phabricator.services.mozilla.com/D261060)
That does look related to the 'native messaging' feature your extension uses.
Some suggestions:
- Still no luck with the ASan build?
- Can you check the Windows Update log to see if it installed updates around the time these crashes became more frequent?
- When does your add-on use the 'native messaging' feature exactly? Can you create a version of the extension that uses it much more frequently to see if that makes the crashes easier to reproduce for us?
It could be some kind of race condition or memory corruption in the native messaging code, but that's just a wild guess.
Comment 17•1 year ago
|
||
(In reply to Jan de Mooij [:jandem] from comment #16)
It could be some kind of race condition or memory corruption in the native messaging code, but that's just a wild guess.
Bug 1986579 comment 9 also vaguely points in this direction (assuming the 1Password app + extension uses the same communication mechanism).
| Reporter | ||
Comment 18•1 year ago
|
||
(In reply to Jan de Mooij [:jandem] from comment #16)
Still no luck with the ASan build?
Will check it soon.
Can you check the Windows Update log to see if it installed updates around the time these crashes became more frequent?
These are the updates from the last month displayed in settings:
Quality Updates:
- 2025–09 Zbirna posodobitev za Windows 11 Version 24H2 za x64 (KB5065426) (26100.6584) - 13. 09. 2025
- 2025-09 Zbirna posodobitev za ogrodji .NET Framework 3.5 in 4.8.1 za Windows 11, version 24H2 za x64 (KB5064401) - 13. 09. 2025
Driver Updates:
- SAMSUNG Electronics Co., Ltd. - USB - 2.21.0.0 - 29. 09. 2025
- SAMSUNG Electronics Co., Ltd. - Modem - 2.21.0.0 - 24. 09. 2025
- SAMSUNG Electronics Co., Ltd. - Net - 2.21.0.0 - 22. 09. 2025
Definition Updates:
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.228.0) – Trenutni kanal (široko) - 30. 09. 2025
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.206.0) – Trenutni kanal (široko) - 29. 09. 2025
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.199.0) – Trenutni kanal (široko) - 28. 09. 2025
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.193.0) – Trenutni kanal (široko) - 27. 09. 2025
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.156.0) – Trenutni kanal (široko) - 26. 09. 2025
- Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.150.0) – Trenutni kanal (široko) - 25. 09. 2025
- More definition updates, almost from every day...
And from the command line:
> wmic qfe list
Caption CSName Description FixComments HotFixID InstallDate InstalledBy InstalledOn Name ServicePackInEffect Status
http://support.microsoft.com/?kbid=5064401 ZALMAN-Z9 Update KB5064401 NT AUTHORITY\SYSTEM 9/13/2025
https://support.microsoft.com/help/5065426 ZALMAN-Z9 Security Update KB5065426 NT AUTHORITY\SYSTEM 9/13/2025
ZALMAN-Z9 Update KB5064531 NT AUTHORITY\SYSTEM 9/13/2025
The first crash happened at 2025-09-28 21:33. Appearently, the only update from that day was the Microsoft Defender definition update. Other recent updates were for some Samsung drivers (probably for phone). Last quality updates were installed almost two weeks before crashes started. Note that I didn't use the computer between 2025-09-14 and 2025-09-20, but I used it in the last week before before crashes.
When does your add-on use the 'native messaging' feature exactly?
On the setup page (setup/install.html), it calls GetSystemVersions every 10 seconds (after the license has been accepted), and InstallRuntime when the user clicks to install the runtime. On the update page (setup/update.html), it also calls GetSystemVersions every 10 seconds.
The background script (background.js) calls GetSystemVersions when the extension is installed (browser.runtime.onInstalled). It also calls GetSystemVersions and GetSiteList on message from the content script, which happens on navigation.
The popup (sites/manage.html) calls GetSystemVersions, GetSiteList and GetProfileList when it's opened, and a few other functions when the user does various actions, like editing or launching a web app.
Can you create a version of the extension that uses it much more frequently to see if that makes the crashes easier to reproduce for us?
I tried to create a minimal extension that just calls GetSystemVersions or GetSiteList every second from the background script, without any other functionality. But it still didn't crash after more than 10 minutes.
One thing that I already tried before was commenting out the whole background and content scripts, and opening only the setup page. This also caused the crashes. So, it seems that native messaging specifically from extension tabs may cause issues. I will try to create some minimal version of that.
Comment 19•1 year ago
|
||
@Filip: I've finally been able to get some crashes locally after I enabled debug mode of the pwa connector app (by creating an empty DEBUG file in the app's data directory). Maybe that slows down the connector app enough that it's more likely to trigger. Maybe we could test a version of the connector app that just sleeps for a few seconds to see if that does the trick also?
| Reporter | ||
Comment 20•1 year ago
|
||
That's a really good find. I actually enabled debug mode on 2025-09-28 21:31, so just a few minutes before crashes started. I now disabled it and crashes seem to be gone. Enabling it back again causes the crash.
What debug mode does is enable debug level logging and redirect internal stdout and stderr to file instead of discarding them.
So maybe it's really because of slowdowns due to additional logging. That stdout and stderr redirections are just to temporarily prevent sending any potential malformed output back to Firefox (for example, from additional processes that the connector spawns), but nothing was written to those files so this probably didn't affect anything.
I'll check what happens if I add a sleep to the program.
| Reporter | ||
Comment 21•1 year ago
|
||
I've added a bunch of 2 second sleeps to the connector code (before and after it reads stdin, when processing, before and after it writes the response, etc., all at the same time), but it still didn't crash. Even with the debug mode enabled and those sleeps in place, it didn't crash, so maybe sleeping for too long somehow fixes the issue.
Then, I removed the sleeps but manually set the log level to debug (keeping stdin/stdout redirections in the normal mode), and it crashed after around a minute of the setup page being opened. The connector log doesn't show anything abnormal (sleep logs are just where the sleeps used to be, without any delay):
12:03:37 [INFO] Received a request: GetSystemVersions(GetSystemVersions)
12:03:37 [WARN] Sleep 2
12:03:37 [INFO] Processed the request: SystemVersions { firefoxpwa: Some("0.0.0"), firefox: Some("143.0.1"), _7zip: Some("25.00") }
12:03:37 [WARN] Sleep 3
12:03:37 [WARN] Sleep 4
12:03:37 [WARN] Sleep 4.1
12:03:37 [WARN] Sleep 4.2
12:03:37 [WARN] Sleep 4.3
12:03:37 [WARN] Sleep 4.4
12:03:37 [WARN] Sleep 4.5
12:03:37 [INFO] Sent a response
Comment 22•1 year ago
|
||
(In reply to Filip Štamcar from comment #21)
Then, I removed the sleeps but manually set the log level to debug (keeping stdin/stdout redirections in the normal mode), and it crashed after around a minute of the setup page being opened.
So that's with just extra debug logging but without the stdout/stderr redirections? Does adding some more log messages affect anything?
A few times I got it to crash immediately but usually it takes longer.
| Reporter | ||
Comment 23•1 year ago
|
||
(In reply to Jan de Mooij [:jandem] from comment #22)
So that's with just extra debug logging but without the stdout/stderr redirections?
Yes. Well, technically, they are redirected to NUL.
In the normal mode (and this custom case), any stdout/stderr, except the final response, is discarded. In the "proper" debug mode, they are redirected to files instead.
Does adding some more log messages affect anything?
As far as I know it shouldn't.
Comment 24•1 year ago
|
||
I suspect this is a problem with the async IO code in subprocess_win.worker.js. Maybe a regression from bug 1979546.
IIUC we give the kernel a pointer to the array buffer's memory here and then let it write into that asynchronously:
this.buffer = new ArrayBuffer(count);
let ok = libc.ReadFile(
this.handle,
this.buffer,
count,
null,
this.overlapped.address()
);
We can clear this.buffer on some code paths and the GC might collect the array buffer after that. Also if count is small, the memory might be allocated inline in the array buffer object, and the GC can move objects in memory.
Updated•1 year ago
|
Comment 25•1 year ago
|
||
(In reply to Jan de Mooij [:jandem] from comment #24)
Nice detective work tracking this down.
ctypes allows array buffers to convert to pointers. There is a comment that talks about potential problems but that won't be seen by people using ctypes from JS. I wonder if there is something we can do to catch problems like this.
https://searchfox.org/firefox-main/source/js/src/ctypes/CTypes.cpp#3539-3558
Comment 26•1 year ago
|
||
(In reply to Jan de Mooij [:jandem] from comment #24)
I suspect this is a problem with the async IO code in
subprocess_win.worker.js. Maybe a regression from bug 1979546.IIUC we give the kernel a pointer to the array buffer's memory here and then let it write into that asynchronously:
this.buffer = new ArrayBuffer(count); let ok = libc.ReadFile( this.handle, this.buffer, count, null, this.overlapped.address() );We can clear
this.bufferon some code paths and the GC might collect the array buffer after that. Also ifcountis small, the memory might be allocated inline in the array buffer object, and the GC can move objects in memory.
What would be a better way of doing this in JS ctypes?
Comment 27•1 year ago
|
||
(In reply to :Gijs (he/him) from comment #26)
What would be a better way of doing this in JS ctypes?
Maybe create a uint8_t array using ctypes (like here?) and then create the ArrayBuffer and copy to it. I'm not very familiar with js-ctypes though but I could check what APIs it has available if needed.
That said, I haven't verified the ArrayBuffer code is what's causing the crashes here. As a first step it'd be good if someone more familiar with the native messaging Web Extension API could create a reliable test case for this bug so we can see what's going on exactly.
| Assignee | ||
Comment 28•1 year ago
|
||
The Subprocess module, a core component of NativeMessaging, was refactored recently in bug 1979546 with a followup in bug 1982950. I speculate that mozregression from comment 15 points to the second bug (which was a regression of the first), because that regression kept the worker thread busy, and that may prevent GC from kicking in and doing stuff that would trigger this crash (this is pure speculation). There was only 2 days between the first patch landing and the second patch landing, so I can easily imagine the first bug to be the regressor but overlooked due to the temporary regression.
Given the correlation of the observed crashes (in crash stats and the mozregression query here) and the known refactoring of bug 1979546, I'm marking that as the regressor of this bug. If further investigation points to a different issue, I'll adjust the queries.
The reporter also commented in bug 1986579 before, associated with the crash signature. I expect that a resolution to this bug would fix the issue reported with bug 1986579, and marked the dependency as such.
These refactors did not fundamentally change the use of ctypes for reading/writing to pipes, but there may be (pre-existing) issues exposed by changes to execution order or code paths.
The ReadFile (and also WriteFile) calls in subprocess_win.worker.js are passed an ArrayBuffer (this.buffer), and that buffer is expected to be kept alive for as long as these methods are busy handling I/O. E.g. ReadFile MSDN documentation states:
This buffer must remain valid for the duration of the read operation. The caller must not use this buffer until the read operation is completed.
I did a full audit of the relevant code, to look for releasing buffers used by ReadFile and WriteFile.
- For
ReadFile's buffer to remain alive as desired, we must not clearbufferuntilReadFileeither returns an error (other thanERROR_IO_PENDING) or signals completion. (similarly forWriteFile). - All cases where
bufferis cleared are as follows (in subprocess/ code) (note:InputPipeis used for stdout/stderr,OutputPipefor stdin,Pipeis their base class):InputPipe'sonReadywhen read completed (similarly forOutputPipe'sonReady, whenWriteFilecompletes). This is OK.Pipe'sclosemethod withforce = true
Their callers are:- close request with
force = trueby a subprocess consumer (via the public commonclosemethod exposed via Subprocess.sys.mjs.
⚠️ (I don't see non-test usages of this fortunately - this code path is susceptible to an early release of the buffer) process.exitPromiseresolution,close(true)is called for stdin (stdin is anOutputPipe), which happens when process exit is detected inProcess'swait().
⚠️ Stdin pipe can outlive the process. (I even wrote a unit test that shows that this can happen in practice).- Pipe's
onError, which in turn has the following callers:InputPipe'smaybeClose- called when process is detected to be closed.- ⚠️ This should verify that winLastError is
ERROR_IO_PENDINGto avoid early release of the buffer
- ⚠️ This should verify that winLastError is
InputPipe'sreadBufferwhenReadFilefails, if the process handle is gone OR the error is notERROR_IO_PENDING- ⚠️ In theory, if the process handle is gone, the pipe could still be valid if the error is ERROR_IO_PENDING. For buffer tracking purposes, we should not ignore the buffer because the pipe can outlive the process.
InputPipe'sonReadywhenGetOverlappedResultfailed- In practice,
pipe.onReadyis only called when completion was signaled through the IOCP, so the buffer does not need to be retained. If we want to be defensive, we could add anERROR_IO_PENDINGcheck there.
- In practice,
OutputPipe'swriteBufferwhenWriteFilefails. Conditional onERROR_IO_PENDING, as desired.InputPipe'sonReadywhenGetOverlappedResultfailed or the number of bytes do not match- In practice,
pipe.onReadyis only called when completion was signaled through the IOCP, so the buffer does not need to be retained. If we want to be defensive, we could add anERROR_IO_PENDINGcheck and handle that case in a special way.
- In practice,
poll()when the pipe reportsERROR_BROKEN_PIPE- I don't expect
ReadFileorWriteFileto be actively engaging with a broken pipe, but that is an assumption on my part.
- I don't expect
poll()when an error is thrown from inpipe.onReady()onReady()should not throw (and would be visible in the console asWorker error: ...).- In practice,
pipe.onReadyis only called when completion was signaled through the IOCP, so the buffer does not need to be retained.
- close request with
I identified some (pre-existing) issues after auditing the code for the scenario of "buffer being eligible for GC sooner than it should be".
For completeness: the refactor changed the handling of process completion in the following significant way:
- Previously, the implementation used
WaitForMultipleObjects, with the objects passed such that signaled pipes and processes were processed in that order (completed pipes first, then the processes). - With IOCP, pipe/process completion statuses are handled in the order as reported by IOCP. So if previously we handled all pipes first and then the process, now we could handle process termination and then continue to process pipes. And maybe trigger one of the pre-existing issues.
Updated•1 year ago
|
| Assignee | ||
Comment 29•1 year ago
|
||
I tried to see if there was anything meaningful in the stack traces; looks like they are all over the place, which is not too surprising since we are looking at memory corruption here.
I think that the best path forwards is to fix the issues I identified in comment 28 based on a code audit.
Thank you for your detailed report Filip. Do you know why you started spotting the issue relatively recently on Nightly? Is there any difference in how the extension uses native messaging, or in how the native messaging host binary exits or interacts via stdin/stdout/stderr?
| Reporter | ||
Comment 30•1 year ago
|
||
(In reply to Rob Wu [:robwu] from comment #29)
Do you know why you started spotting the issue relatively recently on Nightly? Is there any difference in how the extension uses native messaging, or in how the native messaging host binary exits or interacts via stdin/stdout/stderr?
Appearently, because I've only recently enabled the debug mode in the native program of the extension. Basically, the only thing debug mode does is enable debug log level (more details in comment 20 and comment 23), but maybe that slows down the program just enough to trigger the crash. When the minimal log level is set to warning (which is default for the native program), crashes disappear.
I'm not sure if I had debug mode enabled anytime between August and now, but at least no crashes happened (except bp-26a9aca7-af15-4554-92b0-8d7bb0250816), so I guess not.
| Reporter | ||
Comment 31•1 year ago
|
||
Sorry, I somehow removed crash signatures, but I don't know how to add them back.
Comment 32•1 year ago
|
||
The reporter crashed with lot of different signatures, though the most illuminating is EntrySlot<T>::isFree because that has NativeObject.cpp in the stack. That seems to be always a null de-ref. The rest are pretty generic GC signatures as jandem mentioned in bug 1986579, but I'll call out [@ js::gc::detail::ChunkPtrHasStoreBuffer ] because a good portion of the ones experienced by the reporter show signs of a UAF (not surprising in a GC crash). When the full poison value is present in one of the registers, the associated crashing address is either 0xe5e5e5e400000000 or 0xe5e5e5e5e5e00000
sec-moderate rather than sec-high because it requires installing an addon that uses native messaging and appears to be an unreliable race condition. This is probably higher severity for fixing because of the functionality regression than the security aspect.
Comment 33•1 year ago
|
||
I don't know if it's so useful to have the generic signatures in the crash field: it very much obscures the clarity that this is a regression in 143 as seen with just the EmptySlot signature.
Comment 34•1 year ago
|
||
Set release status flags based on info from the regressing bug 1979546
| Assignee | ||
Comment 35•1 year ago
|
||
I don't know if it's so useful to have the generic signatures in the crash field: it very much obscures the clarity that this is a regression in 143 as seen with just the EmptySlot signature.
Across all of the three signatures that I flagged, I see increases mid september. Here are some queries, filtered by release channel=release, platform=Windows, platform=parent to remove as much noise as possible:
- crash graph of
mozilla::detail::EntrySlot<T>::isFree. Goes from barely anything to ~30 a day. - crash graph of
js::gc::detail::ChunkPtrHasStoreBuffer. Goes from 30-50ish a day to about double of that. - crash graph of
js::jit::CacheIRStubInfo::fieldType. Goes from barely anything to 6-22 a day.
If we'd like to see sharper spikes in the generated graph, we can exclude ChunkPtrHasStoreBuffer since it is quite noisy. I will do so, but note that the rates with the remaining crash numbers undersell the severity of the issue.
| Assignee | ||
Comment 36•1 year ago
|
||
| Assignee | ||
Comment 37•1 year ago
|
||
Filip, could you try to verify whether this build fixes the issue? The build is linked from a Try push with the above patch in Treeherder, available after clicking on the "Ba" at "Windows opt" (and/or "Windows debug"), then clicking the "Artifacts and Debugging Tools" tab, as target.zip.
This zip file contains a build of Firefox with the above patch (the link will expire eventually): https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aPpjX-YCQzG3CO9LnsBeog/runs/0/artifacts/public/build/target.zip
| Assignee | ||
Comment 38•1 year ago
|
||
[Tracking Requested - why for this release]:
This is memory corruption issue in the main process on Windows, that sometimes happens when extensions use native messaging (or the devtools Browser Toolbox is used). Probably too late to fix this in 143, but if the above patch is confirmed to resolve the bug, then we should consider including it in 144 (and the matching ESR release, since the patch was also uplifted to ESR140). The patch is designed to fix the issue with minimal risk.
| Reporter | ||
Comment 39•1 year ago
|
||
I can't vertify that build yet, but I will try it later today.
But I just remembered another thing that might be relevant. The native program also sends its log output to stderr. Based on MDN, stderr shouldn't affect the native messaging communication with the extension, but just display in the browser console. So maybe because debug log level was enabled, a lot more than usual was sent to stderr, and sending too much to stderr (and possibly stdin and stdout?) can teigger the crash.
Comment 40•1 year ago
|
||
The Windows debug build from that try push still crashes for me. If it helps, I still see "IOCP notification for unknown pipe: N" debug messages too.
Comment 41•1 year ago
|
||
(In reply to Rob Wu [:robwu] from comment #28)
The
ReadFile(and alsoWriteFile) calls insubprocess_win.worker.jsare passed anArrayBuffer(this.buffer), and that buffer is expected to be kept alive for as long as these methods are busy handling I/O. E.g.ReadFileMSDN documentation states:This buffer must remain valid for the duration of the read operation. The caller must not use this buffer until the read operation is completed.
I did a full audit of the relevant code, to look for releasing buffers used by
ReadFileandWriteFile.
Unfortunately there's more to it than that. ArrayBuffers can store their data inline in the JS object if they are small enough (the max size 96 bytes). Compacting GC can move these objects, invalidating pointers into the data even though buffer remains alive.
If that's what's happening then setting the javascript.options.mem.gc_compacting pref to false should stop the crashes, and using 'Minimize memory usage' in about:memory should help produce one.
| Assignee | ||
Comment 42•1 year ago
|
||
(In reply to Filip Štamcar from comment #39)
But I just remembered another thing that might be relevant. The native program also sends its log output to stderr. Based on MDN, stderr shouldn't affect the native messaging communication with the extension, but just display in the browser console. So maybe because debug log level was enabled, a lot more than usual was sent to stderr, and sending too much to stderr (and possibly stdin and stdout?) can teigger the crash.
We do read stderr and send it to the browser console: https://searchfox.org/firefox-main/rev/08c6e2464d7ac105490e601eec1390e6edef190b/toolkit/components/extensions/NativeMessaging.sys.mjs#302-330
(In reply to Jan de Mooij [:jandem] from comment #40)
The Windows debug build from that try push still crashes for me. If it helps, I still see "IOCP notification for unknown pipe: N" debug messages too.
I'm also able to reproduce. Thanks for describing how to do so in comment 19. I'll clear the needinfo to Filip since we know that the build does not fix the issue.
(In reply to Jon Coppeard (:jonco) from comment #41)
If that's what's happening then setting the
javascript.options.mem.gc_compactingpref to false should stop the crashes, and using 'Minimize memory usage' inabout:memoryshould help produce one.
Still crashes despite setting it to false.
| Assignee | ||
Comment 43•1 year ago
|
||
Filip, could you try to see if this build fixes the issue for you?
The zip file with the build is at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PgApp7CFQKq6SW83reBzBw/runs/0/artifacts/public/build/target.zip
This was linked from the Treeherder entry at: https://treeherder.mozilla.org/jobs?repo=try&revision=8ca496309d47e6da12c3ab00facb675ff3770800&selectedTaskRun=PgApp7CFQKq6SW83reBzBw.0
( I fixed a typo in my previous patch: GetOverlappedResult sets error to ERROR_IO_INCOMPLETE, but I mistakenly wrote ERROR_IO_PENDING instead (because ReadFile and WriteFile use that) ).
Updated•1 year ago
|
| Reporter | ||
Comment 44•1 year ago
|
||
(In reply to Rob Wu [:robwu] from comment #43)
Filip, could you try to see if this build fixes the issue for you?
It seems that build fixes the issue.
| Assignee | ||
Comment 45•1 year ago
•
|
||
Comment on attachment 9518567 [details]
(secure)
Beta/Release Uplift Approval Request
- User impact if declined/Reason for urgency: Windows-only, users of extensions (using nativeMessaging) can experience crashes due to memory corruption in the parent process, caused by race condition between overlapped I/O and garbage collection in the Subprocess module.
- Is this code covered by automated tests?: No
- Has the fix been verified in Nightly?: No
- Needs manual test from QE?: No
- If yes, steps to reproduce: 1. Install https://addons.mozilla.org/firefox/addon/pwas-for-firefox/ (currently version 2.16.0)
- Follow the instructions to install the connector app (e.g.
winget install firefoxpwa --version 2.16.0from Powershell) - Create a file called
DEBUGin%APPDATA%\FirefoxPWA - Wait a bit, maybe up to 10 minutes. Before the patch, Firefox crashes within a few minutes. With the patch, Firefox no longer crashes.
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): Very targeted fix to keep the buffer a bit longer in memory when IO is pending; the expected cleanup is covered by automated unit tests. In the unlikely event that we hold on to the buffer for too long, we have a small memory leak, but that is better than crashing. The reporter has verified that the patch fixes the frequent crash for them (https://bugzilla.mozilla.org/show_bug.cgi?id=1991950#c44).
- String changes made/needed: None
- Is Android affected?: No
| Assignee | ||
Comment 46•1 year ago
|
||
Original Revision: https://phabricator.services.mozilla.com/D267719
Updated•1 year ago
|
Comment 47•1 year ago
|
||
firefox-esr140 Uplift Approval Request
- User impact if declined: Windows-only, users of extensions (using nativeMessaging) can experience crashes due to memory corruption in the parent process, caused by race condition between overlapped I/O and garbage collection in the Subprocess module.
- Code covered by automated testing: no
- Fix verified in Nightly: no
- Needs manual QE test: yes
- Steps to reproduce for manual QE testing: 1. Install https://addons.mozilla.org/firefox/addon/pwas-for-firefox/ (currently version 2.16.0)
- Follow the instructions to install the connector app (e.g. winget install firefoxpwa --version 2.16.0 from Powershell)
- Create a file called DEBUG in
%APPDATA%\FirefoxPWA - Wait a bit, maybe up to 10 minutes. Before the patch, Firefox crashes within a few minutes. With the patch, Firefox no longer crashes.
- Risk associated with taking this patch: low
- Explanation of risk level: Very targeted fix to keep the buffer a bit longer in memory when IO is pending; the expected cleanup is covered by automated unit tests. In the unlikely event that we hold on to the buffer for too long, we have a small memory leak, but that is better than crashing.
Technically the patch is not on Nightly yet, but the reporter has verified that a build with the patch fixes the frequent crash for them (https://bugzilla.mozilla.org/show_bug.cgi?id=1991950#c44).
- String changes made/needed: None
- Is Android affected?: no
Comment 48•1 year ago
|
||
| Assignee | ||
Comment 49•1 year ago
|
||
Note on ESR140: The currently released ESR (ESR140.3.1) is not affected, because I intentionally waited a release (to reduce risk by getting coverage on release) before requesting an uplift of the regressor to ESR140. The planned ESR140.4.0 would be affected, unless the above patch is included in it.
The planned ESR140.4.0 build at https://archive.mozilla.org/pub/firefox/candidates/140.4.0esr-candidates/build1/ is susceptible to this bug, so a new ESR140.4.0 build (respin) is needed to avoid releasing this bug to ESR140.
Comment 50•1 year ago
|
||
Comment 51•1 year ago
|
||
Backed out for causing xpcshell failures @test_subprocess_child.js.
| Assignee | ||
Comment 52•1 year ago
|
||
The test failure is due to this part of the change: https://hg-edge.mozilla.org/integration/autoland/rev/7bcbf9a7f047#l2.96
That logic is called when a process exits, and before my patch the pipe would close immediately. It is more correct to not close immediately (to also match the JSDoc of that function). However, because it is not critical to the fix here, I'll omit it from the patch and fix that issue in a separate bug to minimize the amount of observable changes.
| Assignee | ||
Comment 53•1 year ago
|
||
I moved the fix (that caused the backout due to a test failure) to bug 1993128.
Comment 54•1 year ago
|
||
Comment 55•1 year ago
|
||
Comment 56•1 year ago
|
||
Backed out for causing windows xpcshell failures on test_subprocess.
| Assignee | ||
Comment 57•1 year ago
|
||
The patch added a wait for the IO completion before closing, but if the pipe does not produce or consume the expected number of bytes, then the pipe does not close and the process does not exit. This caused test failures, by the test getting stuck at test_subprocess_huge (and if I skip that test, the test gets stuck at test_subprocess_force_close).
I fixed the problem by adding a CancelIo call to abort the pending IO, and the try push is green: https://treeherder.mozilla.org/jobs?repo=try&revision=5079a011ecfb26678f1f61f381dd490d15aeb3a6
Comment 58•1 year ago
|
||
Comment 59•1 year ago
|
||
The bug is marked as tracked for firefox144 (beta). However, the bug still has low severity.
:jstutte, could you please increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.
For more information, please visit BugBot documentation.
| Assignee | ||
Updated•1 year ago
|
Comment 60•1 year ago
|
||
Comment 61•1 year ago
•
|
||
Hi Filip, Nightly builds with this fix should be available within the next few hours. If you have time, could you please test them out to confirm that everything is working as expected for you with it (build ID 20251008212103 or newer)? Thanks in advance!
Updated•1 year ago
|
Comment 62•1 year ago
|
||
Verified as Fixed. Tested on the latest Nightly (145.0a1/20251008212103) under Windows 11 using the STR from Comment 45.
After ~30 minutes of waiting/using the browser normally the browser does not crash. In comparison, performing the same STR on the latest Release 143 (without the fix) results in a crash 10-15 minutes in.
Comment 63•1 year ago
|
||
The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?
- If yes, please nominate the patch for beta approval.
- See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
- If no, please set
status-firefox144towontfix.
For more information, please visit BugBot documentation.
Comment 64•1 year ago
|
||
Comment on attachment 9518567 [details]
(secure)
Approved for 144.0rc2.
Updated•1 year ago
|
Comment 65•1 year ago
|
||
| uplift | ||
Updated•1 year ago
|
Updated•1 year ago
|
Comment 66•1 year ago
|
||
| uplift | ||
| Assignee | ||
Updated•1 year ago
|
| Reporter | ||
Comment 67•1 year ago
|
||
I can also confirm the crashes no longer happen on Nightly build 20251009092551 after more than 30 minutes of browsing.
Comment 68•1 year ago
|
||
We don't need an ESR-140 advisory because the regressing bug was not checked in until this cycle (see comment #49)
Updated•1 year ago
|
Comment 69•1 year ago
|
||
Verified as Fixed. Tested on the latest Beta (144.0/20251009125714) and ESR (140.4.0esr/20251009121252 from https://treeherder.mozilla.org/jobs?repo=mozilla-esr140&revision=9d719a66ffb784c3d3a34bd64b0550db557eb5dd) under Windows 11 using the STR from Comment 45.
After 50-60 minutes of waiting/using the browser versions normally the browsers did not crash, confirming the fix.
Comment 70•1 year ago
|
||
| Assignee | ||
Comment 71•1 year ago
|
||
:dveditz Could you edit the advisory to emphasize that it is a Windows-only issue? The issue does not apply to Linux or macOS (nor Android, where the nativeMessaging API is not even supported).
Updated•1 year ago
|
| Assignee | ||
Comment 72•1 year ago
•
|
||
I checked the crash stats, and the absence of crashes associated with 144.0 is promising.
Oddly, I see crashes associated with Thunderbird 140.3.1esr despite 140.3.1 never including the regressing patch (it is definitely absent from the 140.3.1 branch, in firefox-main and in comm-central). I also double-checked that the regressing patch is absent from Thunderbird Setup 140.3.1.esr.exe
(edit: I asked gsvelto and according to his analysis, these 140.3.1 crashes associated with the mozilla::detail::EntrySlot<T>::isFree signature can be attributed to bad hardware, raptor lake bugs, externally triggered SIGABRT, which have nothing to do with this bug)
While looking into it, I found that Thunderbird 140.4.0 includes the regression but not the fix from this bug: https://releases.mozilla.org/pub/thunderbird/releases/140.4.0esr/win64/en-US/Thunderbird%20Setup%20140.4.0esr.exe
(to verify: download the above release, and after extraction looked at the modules/subprocess/subprocess_win.worker.js file in omni.ja, and it contains IOCP from bug 1979546 but not "awaitingBufferClose` from this bug.)
Firefox ESR140 releases were never affected because build2 with the fix was created for Firefox140.4 (see comment #49 and comment #68).
But Thunderbird 140.4 branched off FIREFOX_140_4_0esr_BUILD1 instead of BUILD2:
https://hg-edge.mozilla.org/comm-unified/file/THUNDERBIRD_140_4_0esr_RELEASE/.gecko_rev.yml
Because this is now technically a regression in the 140.4 release of TB, TB has to link this advisory in their next release.
| Assignee | ||
Comment 73•11 months ago
|
||
I confirmed that the patch is in Thunderbird 140.5, in https://releases.mozilla.org/pub/thunderbird/releases/140.5.0esr/win64/en-US/Thunderbird%20Setup%20140.5.0esr.exe
Updated•6 months ago
|
Description
•