1794992 - Crash in [@ mio::poll::Poll::poll] calling mio::sys::windows::selector::SelectorInner:: methods

Status: RESOLVED FIXED

(Core :: Audio/Video, defect, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

350 crashes from ~135 installations of Firefox 105.0.x, all Windows, 98.5% with 32-bit builds. There are a few crashes with earlier versions.

These are use-after-frees (address 0xe5e5e5e5).

There are no crash reports for v106 and v107. Matthew, please check if the issue has already been resolved.

Crash report: https://crash-stats.mozilla.org/report/index/afba95a6-2f0e-458d-82e5-572240221013

Reason: EXCEPTION_ACCESS_VIOLATION_EXEC

Top 10 frames of crashing thread:

0 None @0xe5e5e5e5 
1 xul.dll mio::poll::Poll::poll third_party/rust/mio/src/poll.rs:337
2 xul.dll audioipc2::ipccore::EventLoop::poll third_party/rust/audioipc2/src/ipccore.rs:200
3 xul.dll std::sys_common::backtrace::__rust_begin_short_backtrace<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum$<core::result::Result<tuple$<>, std::io::error::Error>, 0, 3, Err> > ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:128
4 xul.dll core::ops::function::FnOnce::call_once<std::thread::impl$0::spawn_unchecked_::closure_env$1<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum$<core::result::Result<tuple$<>, std::io::error::Error>, 0, 3, Err> >, tuple$<> > ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248
5 xul.dll std::sys::windows::thread::impl$0::new::thread_start library/std/src/sys/windows/thread.rs:56
6 kernel32.dll BaseThreadInitThunk 
7 mozglue.dll patched_BaseThreadInitThunk toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:577
8 ntdll.dll _RtlUserThreadStart 
9 ntdll.dll _RtlUserThreadStart 

Comment 1

[Daniel Veditz [:dveditz]]

These are all _EXEC violations, too, so something has really gone off the rails.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

New signature for Windows 7 & 8.1 with Firefox 106.0.

Comment 3

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 20 desktop browser crashes on release (startup)
• Top 10 content process crashes on release

For more information, please visit auto_nag documentation.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

127 crashes with signature [@ mio::sys::windows::selector::SelectorInner::select ] and use-after-free for Firefox 106 and 107 versions, all with 32-bits builds using Windows 8.1 or 10

Crash report: https://crash-stats.mozilla.org/report/index/ecb3e204-51c1-47b6-bb21-c699b0221027

Reason: EXCEPTION_ACCESS_VIOLATION_EXECTop 10 frames of crashing thread:

0  ?  @0xe5e5e5e5  
1  xul.dll  mio::sys::windows::selector::SelectorInner::select  third_party/rust/mio/src/sys/windows/selector.rs:446
1  xul.dll  mio::sys::windows::selector::Selector::select  third_party/rust/mio/src/sys/windows/selector.rs:364
1  xul.dll  mio::poll::Poll::poll  third_party/rust/mio/src/poll.rs:337
2  xul.dll  audioipc2::ipccore::EventLoop::poll  third_party/rust/audioipc2/src/ipccore.rs:201
3  xul.dll  audioipc2::ipccore::impl$7::new::closure$0  third_party/rust/audioipc2/src/ipccore.rs:705
3  xul.dll  std::sys_common::backtrace::__rust_begin_short_backtrace<audioipc2::ipccore::impl$7::new::closure_env$0<audioipc2_client::context::impl$3::init::closure_env$6, audioipc2_client::context::impl$3::init::closure_env$7>, enum$<core::result::Result<tuple$<>, std::io::error::Error>, 0, 3, Err> >  ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:128
4  xul.dll  std::thread::impl$0::spawn_unchecked_::closure$1::closure$0  ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/mod.rs:504
4  xul.dll  core::panic::unwind_safe::impl$23::call_once  ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panic/unwind_safe.rs:271
4  xul.dll  std::panicking::try::do_call  ../4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:492

Comment 5

[Maire Reavy [:mreavy]]

Hey Matthew -- Have you had a chance to look at this? If so, can you provide an update here? Thanks!

Comment 7

[Jim Mathies [:jimm]]

Interesting notes -
_guard_dispatch_icall_nop - fell off the radar in 106.4 for some reason
mio::sys::windows::selector::SelectorInner::select2 only happen in recent esr builds, probably the same select crash

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

_guard_dispatch_icall_nop is being ignored now for the crash signature generation (bug 1798480). The new crash signature is [@ mio::sys::windows::selector::SelectorInner::feed_events] (almost all crashes with Windows 8.1).

Comment 9

[Matthew Gregan [:kinetik]]

Sorry - I should've posted an update on this earlier but was holding off until I had better news. I started investigating on the 25th; so far I've been through multiple crash dumps in windbg and reviewed the AudioIPC code changes that landed since this crash spiked, but don't yet understand the root cause or any possible fixes/mitigations. Also unclear why the crash is centered around Windows 8.1/32-bit systems so far.

Currently attempting to reproduce a crash in a 32-bit Windows 8.1 VM and continuing to investigate other avenues.

Crash analysis/notes:
The crash occurs deep in mio::Poll::poll() after returning from a blocking query (GetQueuedCompletionStatusEx) to the OS for active events; during post-processing each active event has an event-specific callback invoked on it. This callback pointer is stored on the associated event's CompletionStatus object, which is part of a heap-allocated array owned by mio::Events.

AudioIPC's event loop thread owns the mio::Poll and mio::Events in question - these are allocated when the event loop is created (Events with a fixed size), both objects are stored on the event loop thread's stack and freed in the correct order during thread exit when the event loop shuts down. Events is never resized, but is cleared inside mio during each call to poll() , which results in the array of CompletionStatus objects being overwritten by zero-initialized objects.

The abbreviated calling sequence is Poll:poll -> SelectorInner::select (which clears Events) -> select2 -> feed_events, where the crash occurs during callback invocation. Since the clear succeeds, that implies the allocation backing Events.statuses became invalid between select and feed_events. I don't believe this can be caused directly by AudioIPC, since those objects are exclusively owned and used by the event loop thread which is in the Poll::poll call stack at the time of the crash.

There have been a couple of mio issues related to memory management in this area (https://github.com/tokio-rs/mio/pull/1147 and https://github.com/tokio-rs/mio/pull/1154), but Gecko's version of mio (0.8.0) includes those fixes already. There are no obviously related fixes between 0.8.0 and the current version 0.8.5. The NamedPipe support we depend on for AudioIPC has some different code paths than the commonly used Windows network sockets (which use AFD/wepoll internally) - so that's one avenue of continuing investigation.

miow (which provides the IOCP framework for mio on Windows) is currently at 0.3.7 in Gecko (upstream 0.4.0); I'll investigate https://github.com/yoshuawuyts/miow/pull/46 further, but would've expected more frequent issues if that was related. Upgrading mio/miow (and other Rust crates) in Gecko is difficult due to the upstream winapi -> windows-sys transition which is currently blocked in Gecko via bug 1773189, so may need to backport fixes to Gecko's version if it turns out to be an upstream bug.

Gecko started shipping AudioIPC v2 for Windows in Firefox 98. This coincides with a mio change from 0.6 to 0.8, which included major changes to the Windows implementation (switching to AFD/wepoll and rewritten NamedPipe support). There are infrequent crashes going back to at least Firefox 99 (https://crash-stats.mozilla.org/report/index/6286d7e8-9464-4500-8ba7-b41d40220521 and https://crash-stats.mozilla.org/report/index/45c54105-e26e-45bc-b1e2-354f50220624) that appear to be the same issue.

Comment 10

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit auto_nag documentation.

Comment 11

[Jim Mathies [:jimm]]

FYI we're still actively working on this Windows issue.

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

Note: feed_events and select seem to have started in 106 (and are always e5e5 (for feed_events, check the registers). select2 was a ~101-105 crash generally (and are wildptr crashes, not e5e5's). poll is a 100-105 crash (lots of e5e5), nothing since. guard_dispatch_icall_nop is 105-107beta (look like all e5e5). No crashes in 107 release, though, or 108 beta, which is interesting.
These differences between the different crash signatures are interesting.

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

Given the rough stable crash rate, these may be changing signatures between releases for the "same" bug

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

And per :aryx's comment above, guard_dispatch_icall_nop became feed_events, which makes sense.

Comment 16

[Jim Mathies [:jimm]]

Quick notes -

1. we're still diagnosing this bug but don't have a good lead on it yet.
2. crashes are mostly in the content process
3. urls are common video sites
4. vast majority of these crashes happen on Windows 8.1
5. high crash rates per installation
6. comments indicate this is common when transitioning between video segments on sites like youtube

Comment 17

[Jim Mathies [:jimm]]

Technically this bug is stalled at this point. Mathew is going to think of ways we might be able to introduce diagnostic code to help track it down.

Comment 18

[BugBot [:suhaib / :marco]]

The severity field for this bug is set to S3. However, the bug is flagged with the sec-high keyword.
:kinetik, could you consider increasing the severity of this security bug?

For more information, please visit auto_nag documentation.

Comment 19

[Jim Mathies [:jimm]]

Looks to be trailing off, we'll keep an eye on it.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

Rate appears to be stable; maybe it's gone down a smidge

Comment 22

[BugBot [:suhaib / :marco]]

Copying crash signatures from duplicate bugs.

Comment 23

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit BugBot documentation.

Comment 24

[Randell Jesup [:jesup] (needinfo me)]

FYI Jimm, this is still showing 75-100 crashes per day on crash-stats, and is the highest source of e5e5 crashes by a fair margin (10% of all e5e5 crashes in the last month).

Any luck figuring out diagnostics? I think it would be worth investing more effort on that.

Comment 25

[Tom Ritter [:tjr]]

2 of these stacks are still active: select and feed_events. select2 has lots of crashes, but they're all on 102.

• mio::sys::windows::selector::SelectorInner::select and mio::sys::windows::selector::SelectorInner::select2 seem like mostly trying to execute an invalid pointer.
• mio::sys::windows::selector::SelectorInner::feed_events seems mostly trying to read one
• select2 is mostly Windows 10, the other two are mostly 8.1
• mio::sys::windows::selector::SelectorInner::feed_events has a big drop in reports halfway through Sept.
• feed_events and select2 are almost exclusively x64
• select is almost exclusively x86

Comment 26

[Randell Jesup [:jesup] (needinfo me)]

This spiked starting in July. Also the same pattern in 1841741. The total crash rate is concerning even from a stability aspect, but with these being e5e5 it's worse.

Comment 27

[Daniel Veditz [:dveditz]]

Attachment: mio::sys::windows::selector::SelectorInner::select crash history

The crash pattern on these is interesting. the SelectorInner::select crash shows it the most starkly how abruptly the crashes spiked July 1 through Oct 1 and then stopped. Most of the crashes were ESR-115 crashes, but the timing of the spike had nothing to do with versions shipping. That implies it was content-related instead, which makes it even scarier to see these UAF markers. It's basically all 32-bit ESR-115 versions for this specific signature.

Comment 28

[Daniel Veditz [:dveditz]]

The background of crashes includes a lot of French sites. The July 1 to Oct 1 spike has a whole bunch of YouTube in it. Could we ask Google what kind of change they introduced in that period and then stopped? Maybe that would give us a clue as to what triggers this, or even better, get us a video that triggers the crash.

There's also a mio top-crash that started with v130: bug 1916629. That's not exploitable—it's hitting a Rust panic—but it's in a function called from mio::sys::windows::selector::SelectorInner::feed_events() so it's probably related. Did we update mio in 130? Maybe they tried to fix the UAFs here, and partially succeeded in making some of them "safe"? We're using 1.0.1 now and 1.0.2 is available -- maybe there are additional fixes there?

This was marked "stalled" when the incident count was declining. Given this is almost certainly exploitable (if the trigger is found) and clearly affected by web content we should take another look.

Comment 29

[Daniel Veditz [:dveditz]]

The mio::sys::windows::selector::SelectorInner::select signature abruptly dropped in October, from 60-100+ a day to a couple. All look extremely exploitable like comment 4 (except the 3 that have happened in Fx 130-132). The vast majority are ESR-115 (96%), but a reasonable number on ESR-128 (2%). We may have fixed something in 129 or 130, but clearly the drop in crashes for ESR-115 folks is due to web site changes.

The mio::sys::windows::selector::SelectorInner::feed_events signature has cooled down some since October, but still going strong on ESR-115. The ESR-115 crashes (and earlier) are EXCEPTION_ACCESS_VIOLATION_READ on 0xe5e5e5e5e5e5e5e5. The ones in ESR-128 are all EXCEPTION_STACK_BUFFER_OVERRUN / FAST_FAIL_GUARD_ICALL_CHECK_FAILURE with a pointer-looking address.

There are relatively few in 130-133 (16 out of 6000 crashes in the last three months) and they are a mix of the two reasons, but none of the addresses look like a UAF. The EXCEPTION_ACCESS_VIOLATION_READ crashes are on one of these two lines:
https://searchfox.org/mozilla-central/rev/2493c256dbff4e3c7e51a7fc61115df887b87e9e/third_party/rust/mio/src/sys/windows/selector.rs#500,504
The GUARD_ICALL_CHECK crashes are all on this callback line in-between
https://searchfox.org/mozilla-central/rev/2493c256dbff4e3c7e51a7fc61115df887b87e9e/third_party/rust/mio/src/sys/windows/selector.rs#503

from comment 28:

Did we update mio in 130? Maybe they tried to fix the UAFs here, and partially succeeded in making some of them "safe"?

According to bug 1916629 comment 3, yes we did. bug 1910523 and bug 1910513.
Would we backport those to ESR-128? if not we should mark this bug "stalled" again, or wontfix. The crashes in 130+ are not the security bug this was originally filed for.

Comment 30

[Randell Jesup [:jesup] (needinfo me)]

https://crash-stats.mozilla.org/signature/?address=~e5e5e&product=Firefox&date=%3E%3D2025-06-24T14%3A23%3A00.000Z&date=%3C2025-07-01T14%3A23%3A00.000Z&_sort=-date&signature=mio%3A%3Asys%3A%3Awindows%3A%3Aselector%3A%3ASelectorInner%3A%3Afeed_events
is now getting 267 crashes in the last week (top e5e5 crash)

Comment 31

[Daniel Veditz [:dveditz]]

See comment 29: the UAF crashes are "still going strong on ESR-115" (99.4% of the crashes with that signature in the last month, and 100% of the UAF crashes). We're unlikely to fix ESR-115 at this point since we don't know what changed things for later versions

Of the 7 crashes on newer versions in the last month (two in 140 and the rest in ESR-128) the crash reason was the scary EXCEPTION_STACK_BUFFER_OVERRUN / FAST_FAIL_GUARD_ICALL_CHECK_FAILURE. In those 7 cases Windows caught us and slapped us down, but is that all of them or are more subtle stack issues getting through uncaught? We might be doing the wrong thing just as often as in ESR-115 and for some reason it's no longer an allocation that gets poisoned. Or did things get a lot better and maybe this is a different bug in the same code.

Whatever changed this from a UAF to a stack buffer overrun happened before the mio upgrade in 130 because ESR-128 is seeing the bulk of these crashes. Unfortunately we no longer have crash history going back that far.

Comment 32

[Matthew Gregan [:kinetik]]

It's likely this is fixed by bug 2024485, but I'll keep an eye on the crash reports to confirm.

Comment 33

[Daniel Veditz [:dveditz]]

To do that, here's a link to the related crashes without the ESR-115 crashes:
https://crash-stats.mozilla.org/search/?signature=%3D_guard_dispatch_icall_nop&signature=%3Dmio%3A%3Apoll%3A%3APoll%3A%3Apoll&signature=%3Dmio%3A%3Asys%3A%3Awindows%3A%3Aselector%3A%3ASelectorInner%3A%3Afeed_events&signature=%3Dmio%3A%3Asys%3A%3Awindows%3A%3Aselector%3A%3ASelectorInner%3A%3Aselect&signature=%3Dmio%3A%3Asys%3A%3Awindows%3A%3Aselector%3A%3ASelectorInner%3A%3Aselect2&major_version=%3E120&date=%3E%3D2026-01-14T23%3A16%3A00.000Z&_facets=signature&_sort=-version&_sort=-build_id&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=address&_columns=reason#crash-reports

There are still some crash reasons that might be vulnerabilities, but it's a pretty low rate so we'll have to watch for a while.

Comment 34

[Matthew Gregan [:kinetik]]

The UAF pattern this bug originally tracked (0xe5e5e5e5 mozjemalloc poison
reads in mio's Windows selector callback path) is no longer occurring on
supported branches as of the mio 1.1.1 update in bug 2024485 (Firefox 150).

Post-fix data (70 days, 2026-03-19 - 2026-05-28, release channel):

• Zero crashes with 0xe5e5 poison address on any of the tracked
  signatures (feed_events, select, select2, Poll::poll,
  _guard_dispatch_icall_nop).
• Residual feed_events crashes still occur at low rate (56 over 70
  days), but the character is different: 52/56 are CFG-caught
  indirect-call failures (EXCEPTION_STACK_BUFFER_OVERRUN /
  FAST_FAIL_GUARD_ICALL_CHECK_FAILURE), 3 read pointer-shaped
  (non-poison) addresses, 1 EXEC. These are not exploitable in the
  way the original UAF was - Windows safely terminates the process
  when CFG rejects the indirect call.

ESR-115 continues to produce the classic UAF variant but will not be
fixed.

Resolving as FIXED for the original sec-high UAF. Stability follow-up
for the residual CFG-caught crashes filed as bug 2042985.