Closed Bug 1991950 (CVE-2025-11719) Opened 1 year ago Closed 1 year ago

Very frequent crashes when the PWAsForFirefox extension is enabled [extension native messaging]

Categories

(Toolkit :: Async Tooling, defect, P1)

Firefox 145
Unspecified
Windows
defect

Tracking

()

VERIFIED FIXED
145 Branch
Tracking Status
thunderbird_esr140 --- fixed
firefox-esr115 --- unaffected
firefox-esr140 144+ verified
firefox143 --- wontfix
firefox144 + verified
firefox145 + verified

People

(Reporter: filip.stamcar, Assigned: robwu)

References

(Blocks 1 open bug, Regression)

Details

(4 keywords, Whiteboard: [addons-jira][adv-main144+][adv-esr140.4-])

Crash Data

Attachments

(2 files, 2 external files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0

Steps to reproduce:

  1. Install and configure the PWAsForFirefox extension on Windows: https://addons.mozilla.org/firefox/addon/pwas-for-firefox/
  2. Browse like normal: Open tabs, click links, play videos, etc.
  3. At some point the browser just crashes

Actual results:

The browser crashes. The crashes happen seemingly at random, while clicking on links or opening new websites, even while just waiting. Sometimes, multiple crashes happen just in a few minutes, but sometimes, it works normal for quite some time before it crashes.

I'm using Nightly 145.0a1. The crashes started happening for me on 2025-09-28 (or maybe one build before that) and are still happening on 2025-09-30 (when the extension is enabled). These are my recent crash reports:

Crashes have different signatures, but the common ones seem to be:

  • [@ js::gc::detail::ChunkPtrHasStoreBuffer ]
  • [@ mozilla::detail::EntrySlot<T>::isFree ]
  • [@ js::jit::CacheIRStubInfo::fieldType ]

I've initially reported the crash in this comment.

One other user of the extension reported this issue on GitHub. However, for them, the problems started with the update to Firefox 143. These are some of their crash reports:

It appears this happens only on Windows. It happens with PWAsForFirefox 2.16.0 (which was released last week), 2.15.0, and also when using my development build. When the extension is disabled, crashes do not occur.

Expected results:

Firefox shouldn't crash.

OS: Unspecified → Windows

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core

I don't think it's related to video playback. It happens even when no video/audio is playing.

So this extension is causing Firefox to crash frequently: https://github.com/filips123/PWAsForFirefox

Rob, do you have thoughts on this? We don't expose any low-level APIs to extension code right?

Flags: needinfo?(rob)

For an additional data point, I have this addon installed on my linux laptop but I don't have similar crashes.

I haven't been able to reproduce this yet on my Windows machine.

Filip, do you see this with a particular website or any website? Should I change any settings of the add-on after installing it?

It happens regardless of website. When the extension setup wizard is open (in my case, moz-extension://657e0440-4d7d-4abb-add9-32789d0686ce/setup/install.html, not sure if the extension ID is always the same), it seems to always crash in a few minutes. But even when no extension-related pages/popups are open and I'm just browsing websites normally, it also frequently crashes.

The native messaging host part should be installed. Other settings can be default.

Forgot to add, crashes also happens in a fresh profile of the same Firefox Nightly installation with PWAsForFirefox installed after a few minutes of browsing.

I have version 2.16.0 of PWAsForFirefox installed. Is that the same version you have?

Would you be able to try this with a Firefox ASan build? It might give us more information about the crash. Here is a recent Nightly ASan build.

Another option is a Windows debug build. Run firefox.exe from the command line and see if it prints any assertion failures or other errors when it crashes.

Yes, I'm using PWAsForFirefox 2.16.0.

With the ASan build, the crash still didn't happen after around 10 min of browsing. I don't know if this is related, but when I open the extension popup, I see messages like this in the command line:

Worker debug: IOCP notification for unknown pipe: 356
Worker debug: IOCP notification for unknown pipe: 370

With the debug build, the crash still happened. Most recent lines from the log:

[Parent 14636, Main Thread] WARNING: '!lock.ref()->Remove(tuple)', file /builds/worker/checkouts/gecko/toolkit/components/glean/bindings/private/TimingDistribution.cpp:284
[Parent 14636, Main Thread] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x804B000A (NS_ERROR_MALFORMED_URI): file /builds/worker/checkouts/gecko/dom/base/Document.cpp:17407
[Parent 14636, Main Thread] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /builds/worker/checkouts/gecko/parser/html/nsHtml5StreamParser.cpp:1230
[Parent 14636, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:95
Worker debug: IOCP notification for unknown pipe: 2
Worker debug: IOCP notification for unknown pipe: 4
[14636] Assertion failure: chunk->runtime, at /builds/worker/workspace/obj-build/dist/include/js/HeapAPI.h:646
#01: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x84cac2b]
#02: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86515c6]
#03: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8ceb96e]
#04: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86557a6]
#05: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x86553db]
#06: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85ea42b]
#07: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85ea30b]
#08: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x864b0ad]
#09: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x864ab7f]
#10: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e976c]
#11: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e92bf]
#12: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x85e91b7]
#13: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8632cae]
#14: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8638539]
#15: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x863c46e]
#16: XRE_GetBootstrap[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x863dda5]
#17: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2e6ea8e]
#18: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2e6ebca]
#19: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b64f18]
#20: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b645fc]
#21: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x2b74542]
#22: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x9f9cc0]
#23: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x9fa98f]
#24: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa0c138]
#25: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa08b92]
#26: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa08e4a]
#27: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa1ca57]
#28: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa3795c]
#29: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0xa3f9a1]
#30: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x16d4268]
#31: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x164591f]
#32: mozilla_net_is_label_safe[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x16457ef]
#33: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x6e3d8c8]
#34: VR_RuntimePath[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x6f6643e]
#35: GIFFT_LabeledTimingDistributionCancel[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x8152877]
#36: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830a8b8]
#37: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830d753]
#38: CrashReporter::SetRemoteExceptionHandler[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\xul.dll +0x830e440]
#39: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0x17eb]
#40: ???[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0x1247]
#41: TargetNtUnmapViewOfSection[C:\Users\filips\AppData\Local\Temp\nightly-debug\firefox\firefox.exe +0xaf888]
#42: BaseThreadInitThunk[C:\WINDOWS\System32\KERNEL32.DLL +0x2e8d7]

Thanks, that's very helpful.

I'm still not able to reproduce this on my Windows 11 machine. I'll try installing the latest updates...

I'm also going to hide this bug just in case this turns out to be a bad Firefox bug.

Group: javascript-core-security
Component: Audio/Video: Playback → JavaScript Engine

(In reply to Filip Štamcar from comment #6)

The native messaging host part should be installed. Other settings can be default.

Hey, is there some documentation about what the native part does? In particular how does it communicate with the extension?

Flags: needinfo?(filip.stamcar)

Hey, is there some documentation about what the native part does? In particular how does it communicate with the extension?

A high-level description is available on the docs site. It communicates with the extension using browser.runtime.sendNativeMessage. The code for the native part is here (and in other files in the connector directory). In the extension, it's called from various places, for example, here on the setup page to check some program versions.

Flags: needinfo?(filip.stamcar)

Another thing you could try (until one of us is able to reproduce this..) is mozregression, to narrow down when this started happening on the Firefox side.

https://mozilla.github.io/mozregression/

I ran mozregression on range 2024-10-03 - 2025-10-03. I marked the build as good if it didn't crash after around 10 min of browsing. It found the following commit: Bug 1982950 - Stop polling as soon as process exits r=gstoll (https://phabricator.services.mozilla.com/D261060)

This is the progress summary:

Bisecting on mozilla-central [2024-10-03 - 2025-10-03]
Tested mozilla-central build: 2025-04-04 (verdict: g)
Tested mozilla-central build: 2025-07-04 (verdict: g)
Tested mozilla-central build: 2025-08-19 (verdict: b)
Tested mozilla-central build: 2025-07-27 (verdict: g)
Tested mozilla-central build: 2025-08-08 (verdict: g)
Tested mozilla-central build: 2025-08-14 (verdict: b)
Tested mozilla-central build: 2025-08-11 (verdict: g)
Tested mozilla-central build: 2025-08-13 (verdict: g)
Bisecting on mozilla-central [968f8460 - 7de8dcc3]
Tested mozilla-central build: 48334adb (verdict: g)
Bisecting on autoland [48334adb - 7de8dcc3]
Tested autoland build: b270d013 (verdict: g)
Tested autoland build: 172b4f01 (verdict: g)
Tested autoland build: d67fc3a9 (verdict: g)
Tested autoland build: 1387ca28 (verdict: b)
Tested autoland build: 1fad8465 (verdict: g)

I don't know if this is relevant, but on builds b270d0133ee593f14f63d4515b602fba893b9dbc and 172b4f01ef69ad5af1fc8bc8dd2f42193775a4b8, native messaging seems to be broken. Any extension call to browser.runtime.sendNativeMessage failed with the following error:

null ExtensionCommon.sys.mjs:811
    normalizeError resource://gre/modules/ExtensionCommon.sys.mjs:811
    wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:964
    (Async: promise callback)
    wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:921
    wrapPromise resource://gre/modules/ExtensionCommon.sys.mjs:920
    callAsyncFunction resource://gre/modules/ExtensionCommon.sys.mjs:1159
    callAsyncFunction resource://gre/modules/ExtensionChild.sys.mjs:768
    callAndLog resource://gre/modules/ExtensionChild.sys.mjs:748
    callAsyncFunction resource://gre/modules/ExtensionChild.sys.mjs:767
    stub resource://gre/modules/Schemas.sys.mjs:3048
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:714
    (Async: setTimeout handler)
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    checkLicenseAgreement moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:682
    InterpretGeneratorResume self-hosted:1332
    AsyncFunctionNext self-hosted:800
    (Async: async)
    startWizard moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:795
    InterpretGeneratorResume self-hosted:1332
    AsyncFunctionNext self-hosted:800
    (Async: async)
    lMWoA moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:797
    newRequire moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:84
    <anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:144
    <anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:163
Error: An unexpected error occurred
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:714
    setTimeout handler*checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    checkNativeConnection moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:747
    checkLicenseAgreement moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:682
    startWizard moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:795
    lMWoA moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:797
    newRequire moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:84
    <anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:144
    <anonymous> moz-extension://8d48bb9d-7bf4-44bc-bdac-7d53566d03bb/install.c58c8571.js:163
install.c58c8571.js:742:21

I marked these two builds as good as the browser didn't crash. In all other builds, native messaging worked fine (apart from the crash).

It's possible that some of the good builds could still crash after more time. However, on every bad build, the crash happened in less than 5 min and I waited around 10 min, so I think this is unlikely. And the result seems consistent with the bug report of the other extension user that said the crashes started with Firefox 143. However, it's weird that I was using Nightly for the whole time, but the crashes only started for me last week. I have crash report bp-26a9aca7-af15-4554-92b0-8d7bb0250816 from around the time the patch was introduced (2025-08-16) and it has the same signature as some of the newer crashes ([@ js::gc::detail::ChunkPtrHasStoreBuffer ]), but that was the last crash until now, even though I had the extension enabled and also used it.

(In reply to Filip Štamcar from comment #15)

I ran mozregression on range 2024-10-03 - 2025-10-03. I marked the build as good if it didn't crash after around 10 min of browsing. It found the following commit: Bug 1982950 - Stop polling as soon as process exits r=gstoll (https://phabricator.services.mozilla.com/D261060)

That does look related to the 'native messaging' feature your extension uses.

Some suggestions:

  • Still no luck with the ASan build?
  • Can you check the Windows Update log to see if it installed updates around the time these crashes became more frequent?
  • When does your add-on use the 'native messaging' feature exactly? Can you create a version of the extension that uses it much more frequently to see if that makes the crashes easier to reproduce for us?

It could be some kind of race condition or memory corruption in the native messaging code, but that's just a wild guess.

(In reply to Jan de Mooij [:jandem] from comment #16)

It could be some kind of race condition or memory corruption in the native messaging code, but that's just a wild guess.

Bug 1986579 comment 9 also vaguely points in this direction (assuming the 1Password app + extension uses the same communication mechanism).

(In reply to Jan de Mooij [:jandem] from comment #16)

Still no luck with the ASan build?

Will check it soon.

Can you check the Windows Update log to see if it installed updates around the time these crashes became more frequent?

These are the updates from the last month displayed in settings:

Quality Updates:

  • 2025–09 Zbirna posodobitev za Windows 11 Version 24H2 za x64 (KB5065426) (26100.6584) - 13. ‎09. ‎2025
  • 2025-09 Zbirna posodobitev za ogrodji .NET Framework 3.5 in 4.8.1 za Windows 11, version 24H2 za x64 (KB5064401) - 13. ‎09. ‎2025

Driver Updates:

  • SAMSUNG Electronics Co., Ltd. - USB - 2.21.0.0 - 29. ‎09. ‎2025
  • SAMSUNG Electronics Co., Ltd. - Modem - 2.21.0.0 - 24. ‎09. ‎2025
  • SAMSUNG Electronics Co., Ltd. - Net - 2.21.0.0 - 22. ‎09. ‎2025

Definition Updates:

  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.228.0) – Trenutni kanal (široko) - 30. ‎09. ‎2025
  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.206.0) – Trenutni kanal (široko) - 29. ‎09. ‎2025
  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.199.0) – Trenutni kanal (široko) - 28. ‎09. ‎2025
  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.193.0) – Trenutni kanal (široko) - 27. ‎09. ‎2025
  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.156.0) – Trenutni kanal (široko) - 26. ‎09. ‎2025
  • Posodobitev varnostne analize za Microsoft Defender Antivirus – KB2267602 (različica 1.437.150.0) – Trenutni kanal (široko) - ‎25. ‎09. ‎2025
  • More definition updates, almost from every day...

And from the command line:

> wmic qfe list
Caption                                     CSName     Description      FixComments  HotFixID   InstallDate  InstalledBy          InstalledOn  Name  ServicePackInEffect  Status
http://support.microsoft.com/?kbid=5064401  ZALMAN-Z9  Update                        KB5064401               NT AUTHORITY\SYSTEM  9/13/2025
https://support.microsoft.com/help/5065426  ZALMAN-Z9  Security Update               KB5065426               NT AUTHORITY\SYSTEM  9/13/2025
                                            ZALMAN-Z9  Update                        KB5064531               NT AUTHORITY\SYSTEM  9/13/2025

The first crash happened at 2025-09-28 21:33. Appearently, the only update from that day was the Microsoft Defender definition update. Other recent updates were for some Samsung drivers (probably for phone). Last quality updates were installed almost two weeks before crashes started. Note that I didn't use the computer between 2025-09-14 and 2025-09-20, but I used it in the last week before before crashes.

When does your add-on use the 'native messaging' feature exactly?

On the setup page (setup/install.html), it calls GetSystemVersions every 10 seconds (after the license has been accepted), and InstallRuntime when the user clicks to install the runtime. On the update page (setup/update.html), it also calls GetSystemVersions every 10 seconds.

The background script (background.js) calls GetSystemVersions when the extension is installed (browser.runtime.onInstalled). It also calls GetSystemVersions and GetSiteList on message from the content script, which happens on navigation.

The popup (sites/manage.html) calls GetSystemVersions, GetSiteList and GetProfileList when it's opened, and a few other functions when the user does various actions, like editing or launching a web app.

Can you create a version of the extension that uses it much more frequently to see if that makes the crashes easier to reproduce for us?

I tried to create a minimal extension that just calls GetSystemVersions or GetSiteList every second from the background script, without any other functionality. But it still didn't crash after more than 10 minutes.

One thing that I already tried before was commenting out the whole background and content scripts, and opening only the setup page. This also caused the crashes. So, it seems that native messaging specifically from extension tabs may cause issues. I will try to create some minimal version of that.

@Filip: I've finally been able to get some crashes locally after I enabled debug mode of the pwa connector app (by creating an empty DEBUG file in the app's data directory). Maybe that slows down the connector app enough that it's more likely to trigger. Maybe we could test a version of the connector app that just sleeps for a few seconds to see if that does the trick also?

That's a really good find. I actually enabled debug mode on 2025-09-28 21:31, so just a few minutes before crashes started. I now disabled it and crashes seem to be gone. Enabling it back again causes the crash.

What debug mode does is enable debug level logging and redirect internal stdout and stderr to file instead of discarding them.

So maybe it's really because of slowdowns due to additional logging. That stdout and stderr redirections are just to temporarily prevent sending any potential malformed output back to Firefox (for example, from additional processes that the connector spawns), but nothing was written to those files so this probably didn't affect anything.

I'll check what happens if I add a sleep to the program.

I've added a bunch of 2 second sleeps to the connector code (before and after it reads stdin, when processing, before and after it writes the response, etc., all at the same time), but it still didn't crash. Even with the debug mode enabled and those sleeps in place, it didn't crash, so maybe sleeping for too long somehow fixes the issue.

Then, I removed the sleeps but manually set the log level to debug (keeping stdin/stdout redirections in the normal mode), and it crashed after around a minute of the setup page being opened. The connector log doesn't show anything abnormal (sleep logs are just where the sleeps used to be, without any delay):

12:03:37 [INFO] Received a request: GetSystemVersions(GetSystemVersions)
12:03:37 [WARN] Sleep 2
12:03:37 [INFO] Processed the request: SystemVersions { firefoxpwa: Some("0.0.0"), firefox: Some("143.0.1"), _7zip: Some("25.00") }
12:03:37 [WARN] Sleep 3
12:03:37 [WARN] Sleep 4
12:03:37 [WARN] Sleep 4.1
12:03:37 [WARN] Sleep 4.2
12:03:37 [WARN] Sleep 4.3
12:03:37 [WARN] Sleep 4.4
12:03:37 [WARN] Sleep 4.5
12:03:37 [INFO] Sent a response

(In reply to Filip Štamcar from comment #21)

Then, I removed the sleeps but manually set the log level to debug (keeping stdin/stdout redirections in the normal mode), and it crashed after around a minute of the setup page being opened.

So that's with just extra debug logging but without the stdout/stderr redirections? Does adding some more log messages affect anything?

A few times I got it to crash immediately but usually it takes longer.

(In reply to Jan de Mooij [:jandem] from comment #22)

So that's with just extra debug logging but without the stdout/stderr redirections?

Yes. Well, technically, they are redirected to NUL.

In the normal mode (and this custom case), any stdout/stderr, except the final response, is discarded. In the "proper" debug mode, they are redirected to files instead.

Does adding some more log messages affect anything?

As far as I know it shouldn't.

I suspect this is a problem with the async IO code in subprocess_win.worker.js. Maybe a regression from bug 1979546.

IIUC we give the kernel a pointer to the array buffer's memory here and then let it write into that asynchronously:

    this.buffer = new ArrayBuffer(count);
    let ok = libc.ReadFile(
      this.handle,
      this.buffer,
      count,
      null,
      this.overlapped.address()
    );

We can clear this.buffer on some code paths and the GC might collect the array buffer after that. Also if count is small, the memory might be allocated inline in the array buffer object, and the GC can move objects in memory.

Group: javascript-core-security → firefox-core-security
Component: JavaScript Engine → Async Tooling
Product: Core → Toolkit

(In reply to Jan de Mooij [:jandem] from comment #24)
Nice detective work tracking this down.

ctypes allows array buffers to convert to pointers. There is a comment that talks about potential problems but that won't be seen by people using ctypes from JS. I wonder if there is something we can do to catch problems like this.

https://searchfox.org/firefox-main/source/js/src/ctypes/CTypes.cpp#3539-3558

(In reply to Jan de Mooij [:jandem] from comment #24)

I suspect this is a problem with the async IO code in subprocess_win.worker.js. Maybe a regression from bug 1979546.

IIUC we give the kernel a pointer to the array buffer's memory here and then let it write into that asynchronously:

    this.buffer = new ArrayBuffer(count);
    let ok = libc.ReadFile(
      this.handle,
      this.buffer,
      count,
      null,
      this.overlapped.address()
    );

We can clear this.buffer on some code paths and the GC might collect the array buffer after that. Also if count is small, the memory might be allocated inline in the array buffer object, and the GC can move objects in memory.

What would be a better way of doing this in JS ctypes?

Flags: needinfo?(jdemooij)

(In reply to :Gijs (he/him) from comment #26)

What would be a better way of doing this in JS ctypes?

Maybe create a uint8_t array using ctypes (like here?) and then create the ArrayBuffer and copy to it. I'm not very familiar with js-ctypes though but I could check what APIs it has available if needed.

That said, I haven't verified the ArrayBuffer code is what's causing the crashes here. As a first step it'd be good if someone more familiar with the native messaging Web Extension API could create a reliable test case for this bug so we can see what's going on exactly.

Flags: needinfo?(jdemooij)

The Subprocess module, a core component of NativeMessaging, was refactored recently in bug 1979546 with a followup in bug 1982950. I speculate that mozregression from comment 15 points to the second bug (which was a regression of the first), because that regression kept the worker thread busy, and that may prevent GC from kicking in and doing stuff that would trigger this crash (this is pure speculation). There was only 2 days between the first patch landing and the second patch landing, so I can easily imagine the first bug to be the regressor but overlooked due to the temporary regression.

Given the correlation of the observed crashes (in crash stats and the mozregression query here) and the known refactoring of bug 1979546, I'm marking that as the regressor of this bug. If further investigation points to a different issue, I'll adjust the queries.

The reporter also commented in bug 1986579 before, associated with the crash signature. I expect that a resolution to this bug would fix the issue reported with bug 1986579, and marked the dependency as such.


These refactors did not fundamentally change the use of ctypes for reading/writing to pipes, but there may be (pre-existing) issues exposed by changes to execution order or code paths.

The ReadFile (and also WriteFile) calls in subprocess_win.worker.js are passed an ArrayBuffer (this.buffer), and that buffer is expected to be kept alive for as long as these methods are busy handling I/O. E.g. ReadFile MSDN documentation states:

This buffer must remain valid for the duration of the read operation. The caller must not use this buffer until the read operation is completed.

I did a full audit of the relevant code, to look for releasing buffers used by ReadFile and WriteFile.

I identified some (pre-existing) issues after auditing the code for the scenario of "buffer being eligible for GC sooner than it should be".

For completeness: the refactor changed the handling of process completion in the following significant way:

  • Previously, the implementation used WaitForMultipleObjects, with the objects passed such that signaled pipes and processes were processed in that order (completed pipes first, then the processes).
  • With IOCP, pipe/process completion statuses are handled in the order as reported by IOCP. So if previously we handled all pipes first and then the process, now we could handle process termination and then continue to process pipes. And maybe trigger one of the pre-existing issues.
Assignee: nobody → rob
Blocks: 1986579
Severity: -- → S3
Status: UNCONFIRMED → ASSIGNED
Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ]
Ever confirmed: true
Flags: needinfo?(rob)
Keywords: regression
Priority: -- → P1
Whiteboard: [addons-jira]
Summary: Very frequent crashes when the PWAsForFirefox extension is enabled → Very frequent crashes when the PWAsForFirefox extension is enabled [extension native messaging]

I tried to see if there was anything meaningful in the stack traces; looks like they are all over the place, which is not too surprising since we are looking at memory corruption here.

I think that the best path forwards is to fix the issues I identified in comment 28 based on a code audit.

Thank you for your detailed report Filip. Do you know why you started spotting the issue relatively recently on Nightly? Is there any difference in how the extension uses native messaging, or in how the native messaging host binary exits or interacts via stdin/stdout/stderr?

Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ] → [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ] [@ js::jit::CacheIRStubInfo::fieldType ]

(In reply to Rob Wu [:robwu] from comment #29)

Do you know why you started spotting the issue relatively recently on Nightly? Is there any difference in how the extension uses native messaging, or in how the native messaging host binary exits or interacts via stdin/stdout/stderr?

Appearently, because I've only recently enabled the debug mode in the native program of the extension. Basically, the only thing debug mode does is enable debug log level (more details in comment 20 and comment 23), but maybe that slows down the program just enough to trigger the crash. When the minimal log level is set to warning (which is default for the native program), crashes disappear.

I'm not sure if I had debug mode enabled anytime between August and now, but at least no crashes happened (except bp-26a9aca7-af15-4554-92b0-8d7bb0250816), so I guess not.

Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ] [@ js::jit::CacheIRStubInfo::fieldType ]

Sorry, I somehow removed crash signatures, but I don't know how to add them back.

The reporter crashed with lot of different signatures, though the most illuminating is EntrySlot<T>::isFree because that has NativeObject.cpp in the stack. That seems to be always a null de-ref. The rest are pretty generic GC signatures as jandem mentioned in bug 1986579, but I'll call out [@ js::gc::detail::ChunkPtrHasStoreBuffer ] because a good portion of the ones experienced by the reporter show signs of a UAF (not surprising in a GC crash). When the full poison value is present in one of the registers, the associated crashing address is either 0xe5e5e5e400000000 or 0xe5e5e5e5e5e00000

sec-moderate rather than sec-high because it requires installing an addon that uses native messaging and appears to be an unreliable race condition. This is probably higher severity for fixing because of the functionality regression than the security aspect.

Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ]
Regressed by: 1979546
See Also: → 1982950

I don't know if it's so useful to have the generic signatures in the crash field: it very much obscures the clarity that this is a regression in 143 as seen with just the EmptySlot signature.

Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ] → [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ] [@ js::jit::CacheIRStubInfo::fieldType ]

Set release status flags based on info from the regressing bug 1979546

I don't know if it's so useful to have the generic signatures in the crash field: it very much obscures the clarity that this is a regression in 143 as seen with just the EmptySlot signature.

Across all of the three signatures that I flagged, I see increases mid september. Here are some queries, filtered by release channel=release, platform=Windows, platform=parent to remove as much noise as possible:

If we'd like to see sharper spikes in the generated graph, we can exclude ChunkPtrHasStoreBuffer since it is quite noisy. I will do so, but note that the rates with the remaining crash numbers undersell the severity of the issue.

Crash Signature: [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::gc::detail::ChunkPtrHasStoreBuffer ] [@ js::jit::CacheIRStubInfo::fieldType ] → [@ mozilla::detail::EntrySlot<T>::isFree ] [@ js::jit::CacheIRStubInfo::fieldType ]
Attached file (secure) —

Filip, could you try to verify whether this build fixes the issue? The build is linked from a Try push with the above patch in Treeherder, available after clicking on the "Ba" at "Windows opt" (and/or "Windows debug"), then clicking the "Artifacts and Debugging Tools" tab, as target.zip.

This zip file contains a build of Firefox with the above patch (the link will expire eventually): https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aPpjX-YCQzG3CO9LnsBeog/runs/0/artifacts/public/build/target.zip

Flags: needinfo?(filip.stamcar)

[Tracking Requested - why for this release]:

This is memory corruption issue in the main process on Windows, that sometimes happens when extensions use native messaging (or the devtools Browser Toolbox is used). Probably too late to fix this in 143, but if the above patch is confirmed to resolve the bug, then we should consider including it in 144 (and the matching ESR release, since the patch was also uplifted to ESR140). The patch is designed to fix the issue with minimal risk.

I can't vertify that build yet, but I will try it later today.

But I just remembered another thing that might be relevant. The native program also sends its log output to stderr. Based on MDN, stderr shouldn't affect the native messaging communication with the extension, but just display in the browser console. So maybe because debug log level was enabled, a lot more than usual was sent to stderr, and sending too much to stderr (and possibly stdin and stdout?) can teigger the crash.

The Windows debug build from that try push still crashes for me. If it helps, I still see "IOCP notification for unknown pipe: N" debug messages too.

(In reply to Rob Wu [:robwu] from comment #28)

The ReadFile (and also WriteFile) calls in subprocess_win.worker.js are passed an ArrayBuffer (this.buffer), and that buffer is expected to be kept alive for as long as these methods are busy handling I/O. E.g. ReadFile MSDN documentation states:

This buffer must remain valid for the duration of the read operation. The caller must not use this buffer until the read operation is completed.

I did a full audit of the relevant code, to look for releasing buffers used by ReadFile and WriteFile.

Unfortunately there's more to it than that. ArrayBuffers can store their data inline in the JS object if they are small enough (the max size 96 bytes). Compacting GC can move these objects, invalidating pointers into the data even though buffer remains alive.

If that's what's happening then setting the javascript.options.mem.gc_compacting pref to false should stop the crashes, and using 'Minimize memory usage' in about:memory should help produce one.

(In reply to Filip Štamcar from comment #39)

But I just remembered another thing that might be relevant. The native program also sends its log output to stderr. Based on MDN, stderr shouldn't affect the native messaging communication with the extension, but just display in the browser console. So maybe because debug log level was enabled, a lot more than usual was sent to stderr, and sending too much to stderr (and possibly stdin and stdout?) can teigger the crash.

We do read stderr and send it to the browser console: https://searchfox.org/firefox-main/rev/08c6e2464d7ac105490e601eec1390e6edef190b/toolkit/components/extensions/NativeMessaging.sys.mjs#302-330

(In reply to Jan de Mooij [:jandem] from comment #40)

The Windows debug build from that try push still crashes for me. If it helps, I still see "IOCP notification for unknown pipe: N" debug messages too.

I'm also able to reproduce. Thanks for describing how to do so in comment 19. I'll clear the needinfo to Filip since we know that the build does not fix the issue.

(In reply to Jon Coppeard (:jonco) from comment #41)

If that's what's happening then setting the javascript.options.mem.gc_compacting pref to false should stop the crashes, and using 'Minimize memory usage' in about:memory should help produce one.

Still crashes despite setting it to false.

Flags: needinfo?(filip.stamcar)

Filip, could you try to see if this build fixes the issue for you?

The zip file with the build is at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PgApp7CFQKq6SW83reBzBw/runs/0/artifacts/public/build/target.zip

This was linked from the Treeherder entry at: https://treeherder.mozilla.org/jobs?repo=try&revision=8ca496309d47e6da12c3ab00facb675ff3770800&selectedTaskRun=PgApp7CFQKq6SW83reBzBw.0

( I fixed a typo in my previous patch: GetOverlappedResult sets error to ERROR_IO_INCOMPLETE, but I mistakenly wrote ERROR_IO_PENDING instead (because ReadFile and WriteFile use that) ).

Flags: needinfo?(filip.stamcar)

(In reply to Rob Wu [:robwu] from comment #43)

Filip, could you try to see if this build fixes the issue for you?

It seems that build fixes the issue.

Flags: needinfo?(filip.stamcar)

Comment on attachment 9518567 [details]
(secure)

Beta/Release Uplift Approval Request

  • User impact if declined/Reason for urgency: Windows-only, users of extensions (using nativeMessaging) can experience crashes due to memory corruption in the parent process, caused by race condition between overlapped I/O and garbage collection in the Subprocess module.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce: 1. Install https://addons.mozilla.org/firefox/addon/pwas-for-firefox/ (currently version 2.16.0)
  1. Follow the instructions to install the connector app (e.g. winget install firefoxpwa --version 2.16.0 from Powershell)
  2. Create a file called DEBUG in %APPDATA%\FirefoxPWA
  3. Wait a bit, maybe up to 10 minutes. Before the patch, Firefox crashes within a few minutes. With the patch, Firefox no longer crashes.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Very targeted fix to keep the buffer a bit longer in memory when IO is pending; the expected cleanup is covered by automated unit tests. In the unlikely event that we hold on to the buffer for too long, we have a small memory leak, but that is better than crashing. The reporter has verified that the patch fixes the frequent crash for them (https://bugzilla.mozilla.org/show_bug.cgi?id=1991950#c44).
  • String changes made/needed: None
  • Is Android affected?: No
Attachment #9518567 - Flags: approval-mozilla-release?
Attached file (secure) —
Attachment #9518761 - Flags: approval-mozilla-esr140?

firefox-esr140 Uplift Approval Request

  • User impact if declined: Windows-only, users of extensions (using nativeMessaging) can experience crashes due to memory corruption in the parent process, caused by race condition between overlapped I/O and garbage collection in the Subprocess module.
  • Code covered by automated testing: no
  • Fix verified in Nightly: no
  • Needs manual QE test: yes
  • Steps to reproduce for manual QE testing: 1. Install https://addons.mozilla.org/firefox/addon/pwas-for-firefox/ (currently version 2.16.0)
  1. Follow the instructions to install the connector app (e.g. winget install firefoxpwa --version 2.16.0 from Powershell)
  2. Create a file called DEBUG in %APPDATA%\FirefoxPWA
  3. Wait a bit, maybe up to 10 minutes. Before the patch, Firefox crashes within a few minutes. With the patch, Firefox no longer crashes.
  • Risk associated with taking this patch: low
  • Explanation of risk level: Very targeted fix to keep the buffer a bit longer in memory when IO is pending; the expected cleanup is covered by automated unit tests. In the unlikely event that we hold on to the buffer for too long, we have a small memory leak, but that is better than crashing.

Technically the patch is not on Nightly yet, but the reporter has verified that a build with the patch fixes the frequent crash for them (https://bugzilla.mozilla.org/show_bug.cgi?id=1991950#c44).

  • String changes made/needed: None
  • Is Android affected?: no
Flags: qe-verify+

Note on ESR140: The currently released ESR (ESR140.3.1) is not affected, because I intentionally waited a release (to reduce risk by getting coverage on release) before requesting an uplift of the regressor to ESR140. The planned ESR140.4.0 would be affected, unless the above patch is included in it.

The planned ESR140.4.0 build at https://archive.mozilla.org/pub/firefox/candidates/140.4.0esr-candidates/build1/ is susceptible to this bug, so a new ESR140.4.0 build (respin) is needed to avoid releasing this bug to ESR140.

Pushed by agoloman@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/229c3c661fae https://hg.mozilla.org/integration/autoland/rev/3028a7ae64c3 Revert "Bug 1991950 - Hold onto buffer while IO is pending r=gstoll" for causing xpcshell failures @test_subprocess_child.js.

Backed out for causing xpcshell failures @test_subprocess_child.js.

Flags: needinfo?(rob)

The test failure is due to this part of the change: https://hg-edge.mozilla.org/integration/autoland/rev/7bcbf9a7f047#l2.96
That logic is called when a process exits, and before my patch the pipe would close immediately. It is more correct to not close immediately (to also match the JSDoc of that function). However, because it is not critical to the fix here, I'll omit it from the patch and fix that issue in a separate bug to minimize the amount of observable changes.

Flags: needinfo?(rob)

I moved the fix (that caused the backout due to a test failure) to bug 1993128.

See Also: → 1993128
Pushed by csabou@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/d24e272614b0 https://hg.mozilla.org/integration/autoland/rev/2ca1eeb1d38c Revert "Bug 1991950 - Hold onto buffer while IO is pending r=gstoll" for causing windows xpcshell failures on test_subprocess.

Backed out for causing windows xpcshell failures on test_subprocess.

Push with failures

Failure log

Backout link

Flags: needinfo?(rob)

The patch added a wait for the IO completion before closing, but if the pipe does not produce or consume the expected number of bytes, then the pipe does not close and the process does not exit. This caused test failures, by the test getting stuck at test_subprocess_huge (and if I skip that test, the test gets stuck at test_subprocess_force_close).

I fixed the problem by adding a CancelIo call to abort the pending IO, and the try push is green: https://treeherder.mozilla.org/jobs?repo=try&revision=5079a011ecfb26678f1f61f381dd490d15aeb3a6

Flags: needinfo?(rob)

The bug is marked as tracked for firefox144 (beta). However, the bug still has low severity.

:jstutte, could you please increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Flags: needinfo?(jstutte)
Severity: S3 → S2
Flags: needinfo?(jstutte)
Group: firefox-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 145 Branch

Hi Filip, Nightly builds with this fix should be available within the next few hours. If you have time, could you please test them out to confirm that everything is working as expected for you with it (build ID 20251008212103 or newer)? Thanks in advance!

Flags: needinfo?(filip.stamcar)

Verified as Fixed. Tested on the latest Nightly (145.0a1/20251008212103) under Windows 11 using the STR from Comment 45.

After ~30 minutes of waiting/using the browser normally the browser does not crash. In comparison, performing the same STR on the latest Release 143 (without the fix) results in a crash 10-15 minutes in.

Status: RESOLVED → VERIFIED

The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?

For more information, please visit BugBot documentation.

Flags: needinfo?(rob)

Comment on attachment 9518567 [details]
(secure)

Approved for 144.0rc2.

Attachment #9518567 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #9518761 - Flags: approval-mozilla-esr140? → approval-mozilla-esr140+
Flags: needinfo?(rob)

I can also confirm the crashes no longer happen on Nightly build 20251009092551 after more than 30 minutes of browsing.

Flags: needinfo?(filip.stamcar)

We don't need an ESR-140 advisory because the regressing bug was not checked in until this cycle (see comment #49)

Whiteboard: [addons-jira] → [addons-jira][adv-main144+][adv-esr140.4-]
QA Whiteboard: [sec] [qa-triage-done-c145/b144]

Verified as Fixed. Tested on the latest Beta (144.0/20251009125714) and ESR (140.4.0esr/20251009121252 from https://treeherder.mozilla.org/jobs?repo=mozilla-esr140&revision=9d719a66ffb784c3d3a34bd64b0550db557eb5dd) under Windows 11 using the STR from Comment 45.

After 50-60 minutes of waiting/using the browser versions normally the browsers did not crash, confirming the fix.

:dveditz Could you edit the advisory to emphasize that it is a Windows-only issue? The issue does not apply to Linux or macOS (nor Android, where the nativeMessaging API is not even supported).

Alias: CVE-2025-11719

I checked the crash stats, and the absence of crashes associated with 144.0 is promising.
Oddly, I see crashes associated with Thunderbird 140.3.1esr despite 140.3.1 never including the regressing patch (it is definitely absent from the 140.3.1 branch, in firefox-main and in comm-central). I also double-checked that the regressing patch is absent from Thunderbird Setup 140.3.1.esr.exe
(edit: I asked gsvelto and according to his analysis, these 140.3.1 crashes associated with the mozilla::detail::EntrySlot<T>::isFree signature can be attributed to bad hardware, raptor lake bugs, externally triggered SIGABRT, which have nothing to do with this bug)

While looking into it, I found that Thunderbird 140.4.0 includes the regression but not the fix from this bug: https://releases.mozilla.org/pub/thunderbird/releases/140.4.0esr/win64/en-US/Thunderbird%20Setup%20140.4.0esr.exe
(to verify: download the above release, and after extraction looked at the modules/subprocess/subprocess_win.worker.js file in omni.ja, and it contains IOCP from bug 1979546 but not "awaitingBufferClose` from this bug.)

Firefox ESR140 releases were never affected because build2 with the fix was created for Firefox140.4 (see comment #49 and comment #68).
But Thunderbird 140.4 branched off FIREFOX_140_4_0esr_BUILD1 instead of BUILD2:
https://hg-edge.mozilla.org/comm-unified/file/THUNDERBIRD_140_4_0esr_RELEASE/.gecko_rev.yml

Because this is now technically a regression in the 140.4 release of TB, TB has to link this advisory in their next release.

Group: core-security-release
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: