2068899 - startup Crash in [@ @0x0 | neqo_udp::recv_inner]

Status: RESOLVED FIXED

(Core :: Networking, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Crash signature exists since Firefox 153. ~120 crash reports from ~100 installs of Firefox 154.0 during its release cycle of two weeks.

Crash report: https://crash-stats.mozilla.org/report/index/ff3aa67b-f4c7-4e4c-9699-3a8cc0260903

Reason:

EXCEPTION_ACCESS_VIOLATION_EXEC

Top 10 frames:

0  ?  @0x0000000000000000
1  ?  @0x00007ffb00000000
2  xul.dll  neqo_udp::recv_inner(enum2$<core::net::socket_addr::SocketAddr>, quinn_udp::i...  third_party/rust/neqo-udp/src/lib.rs:158
2  xul.dll  neqo_udp::Socket<std::os::windows::io::socket::BorrowedSocket>::recv(enum2$<c...  third_party/rust/neqo-udp/src/lib.rs:288
2  xul.dll  neqo_glue::neqo_http3conn_process_input::closure$0(neqo_glue::neqo_http3conn_...  netwerk/socket/neqo_glue/src/lib.rs:1211
2  xul.dll  std::thread::local::impl$6::with_borrow_mut::closure$0(std::thread::local::im...  /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/local.rs:757
2  xul.dll  std::thread::local::LocalKey<core::cell::RefCell<neqo_udp::RecvBuf> >::try_wi...  /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/local.rs:513
2  xul.dll  std::thread::local::LocalKey<core::cell::RefCell<neqo_udp::RecvBuf> >::with(s...  /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/local.rs:477
2  xul.dll  std::thread::local::LocalKey<core::cell::RefCell<neqo_udp::RecvBuf> >::with_b...  /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/local.rs:757
2  xul.dll  neqo_glue::neqo_http3conn_process_input(neqo_glue::NeqoHttp3Conn*)  netwerk/socket/neqo_glue/src/lib.rs:1205

Comment 1

[Valentin Gosu [:valentin] {{PTO until Nov 1st}}]

Max, could you take a look?

Comment 2

[Lars Eggert [:lars]]

All Windows, it seems.

Comment 3

[Lars Eggert [:lars]]

Diagnosis

Windows-only parent-process ACCESS_VIOLATION_EXEC at 0x0 on the Socket Thread. 78 reports / 56 installs in the 154 cycle; 60 have uptime <5s and 54 crashed again within 300s — affected users are in a crash loop.

Sibling signature @0x0 | sc2wsp.dll | quinn_udp::imp::UdpSocketState::recv (152.0.6, 77fee0ce-9333-4252-b31b-598390260828) shows the mechanism directly: three SC2WSP.dll frames between quinn-udp/src/windows.rs:258 — the indirect call (wsa_recvmsg_ptr)(...) — and the jump to null. SC2WSP.dll is symbol-less, so on the main signature the walker loses those frames to scan.

SC2WSP.dll + SC2ClientCP.dll (cert AT "IIT") appear in 20/20 sampled reports vs 0/20 baseline Windows/154.0.1 parent crashes. Locales uk 55 / ru 15 / en-US 7.

The software. SC2WSP.dll v2.0.1.6 is the connection-protection library of ІІТ Захист з'єднань-2. Клієнт (АТ «ІІТ», Kharkiv), which sets up DSTU-encrypted sessions to Ukrainian state e-services; registries like inforesurs.gov.ua publish install guides, so affected users can't just uninstall it. Signed and clean, loaded non-relocated at 0x180000000. It registers as a Winsock 2 Layered Service Provider — a DLL in the Winsock catalog sitting between the app and the TCP/IP stack, so every socket call, including WSAIoctl(SIO_GET_EXTENSION_FUNCTION_POINTER), is serviced by it first. Microsoft deprecated LSPs in favour of WFP, but Win32 apps like Firefox remain subject to them.

Root cause on our side: third_party/rust/quinn-udp/src/windows.rs:512 resolves WSARecvMsg once per process on a throwaway AF_INET socket that it then closes, and calls that pointer on every other socket. SIO_GET_EXTENSION_FUNCTION_POINTER returns a provider-specific pointer. Firefox's real QUIC socket comes from NSPR (w95sock.c:52) and may be AF_INET6, so with an LSP in the catalog the pointer can belong to a different provider chain than the socket it is used on. The LSP thunk then dereferences a null per-socket slot. In-tree code that does this correctly resolves per-socket: netwerk/sctp/src/user_recv_thread.c:517, nsprpub/.../w95sock.c:342.

Fix

In third_party/rust/quinn-udp/src/windows.rs:

• Drop the WSARECVMSG_PTR LazyLock and its probe socket.
• Add wsa_recvmsg: WinSock::LPFN_WSARECVMSG to UdpSocketState, resolved in UdpSocketState::new() against socket.0. Keep returning ErrorKind::Unsupported on failure, replacing the is_none() check at line 76.
• recv() (line 232) reads self.wsa_recvmsg; drop the .expect(...).

Same direction as quinn#2491 (global LazyLock state that belongs on UdpSocketState). No upstream issue covers WSARecvMsg — file one, land the PR, cut a release, ./mach vendor rust.

If the LSP also refuses the ioctl on the real socket, neqo_udp::Socket::new fails at netwerk/socket/neqo_glue/src/lib.rs:413 and Necko falls back — degradation, not a crash.

Comment 4

[Lars Eggert [:lars]]

Windows security software strikes again. Yay.

Comment 5

[Lars Eggert [:lars]]

https://github.com/quinn-rs/quinn/pull/2825

Comment 6

[Max Inden]

Bug 2069696 landed. Sebastian can you still see the crashes on Nightly?

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

No crashes for version 157 and 158 at this point.

Interestingly, ESR153 has only crash reports for version 153.0esr but not for newer versions while corresponding "fast" release branches 154 and 155 have crash reports. Should there be an uplift request of bug 2069696 to ESR153 despite the crashes having stopped without it?

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Please request ESR153 uplift on this when you get a chance.

Comment 9

[Lars Eggert [:lars]]

The fix is actually in Bug 2069696, requesting uplift there.