1614305 - Crash in [@ IPCError-browser | ShutDownKill | NtYieldExecution]

Status: RESOLVED DUPLICATE of bug 1279293

(Core :: XPCOM, defect, P3)

Description

[Pascal Chevrel:pascalc]

This bug is for crash report bp-d90d2a48-9fa2-41a9-800d-3df1c0200210.

Top 10 frames of crashing thread:

0 ntdll.dll NtYieldExecution 
1 user32.dll PeekMessageW 
2 xul.dll SingleNativeEventPump::OnProcessNextEvent widget/windows/nsAppShell.cpp:140
3 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1124
4 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:486
5 xul.dll mozilla::ipc::MessagePump::Run ipc/glue/MessagePump.cpp:87
6 xul.dll MessageLoop::RunHandler ipc/chromium/src/base/message_loop.cc:308
7 xul.dll MessageLoop::Run ipc/chromium/src/base/message_loop.cc:290
8 xul.dll nsBaseAppShell::Run widget/nsBaseAppShell.cpp:137
9 xul.dll nsAppShell::Run widget/windows/nsAppShell.cpp:406

Comment 1

[Gabriele Svelto [:gsvelto]]

This signature shows up only on nightly which is odd, because I don't see why it shouldn't show up in release build. What makes it worrisome is that if you look at the crashes you'll see that all the threads are stopped waiting. The content processes here didn't even begin shutting down, they're just sitting idle.

Comment 2

[Pascal Chevrel:pascalc]

Tracking given the volume, let's see if the crashes get into 74 beta.

Comment 3

[Andrew McCreight [:mccr8]]

Spinning the event loop is sort of like XPCOM...

Comment 4

[Pascal Chevrel:pascalc]

Restricted to nightly, no crashes in 74 beta so the release is unaffected.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 6

[Ethan Tseng [:ethan]]

Nathan, the crash volume seems to be high. Could you set the priority flag for this bug?

Comment 7

[Nathan Froyd [:froydnj]]

I looked at the first seven crashes that came up with chkimg:

froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/59777fd9-c883-436a-a2b0-f02000200303.dmp 
info: looking for ntdll.dll at https://symbols.mozilla.org/...
info: looking for ntdll.dll at https://msdl.microsoft.com/download/symbols...
info: fetching ntdll.dll from https://msdl.microsoft.com/download/symbols...
crashing IP: 0x7ff81c5bc984
mismatch: 0x7ff81c5bc250 .. 0x7ff81c5bc25b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c5 3e a6 f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ff81c5bc850 .. 0x7ff81c5bc85b (12 bytes) in ntdll.dll
  [ 48 b8 50 c7 3e a6 f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ff81c5bcb50 .. 0x7ff81c5bcb5b (12 bytes) in ntdll.dll
  [ 48 b8 80 c6 3e a6 f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/806691b0-abd6-4f8c-b7a2-7cec80200303.dmp 
crashing IP: 0x7ffb732dc984
mismatch: 0x7ffb732dc250 .. 0x7ffb732dc25b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c5 58 7f f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffb732dc710 .. 0x7ffb732dc71b (12 bytes) in ntdll.dll
  [ 48 b8 10 c7 58 7f f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 33 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffb732dc850 .. 0x7ffb732dc85b (12 bytes) in ntdll.dll
  [ 48 b8 50 c7 58 7f f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffb732dcb50 .. 0x7ffb732dcb5b (12 bytes) in ntdll.dll
  [ 48 b8 80 c6 58 7f f6 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/26de624f-30e5-4ec4-9595-2fa500200303.dmp 
crashing IP: 0x7ffd4a4bc984
mismatch: 0x7ffd4a4bc250 .. 0x7ffd4a4bc25b (12 bytes) in ntdll.dll
  [ 48 b8 b0 c7 b9 ba f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffd4a4bc850 .. 0x7ffd4a4bc85b (12 bytes) in ntdll.dll
  [ 48 b8 20 c9 b9 ba f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffd4a4bcb50 .. 0x7ffd4a4bcb5b (12 bytes) in ntdll.dll
  [ 48 b8 50 c8 b9 ba f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/72653df0-93f7-42f5-864b-1dc190200303.dmp 
crashing IP: 0x7ffc0255c984
mismatch: 0x7ffc0255c250 .. 0x7ffc0255c25b (12 bytes) in ntdll.dll
  [ 48 b8 70 c5 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffc0255c850 .. 0x7ffc0255c85b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c6 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffc0255cb50 .. 0x7ffc0255cb5b (12 bytes) in ntdll.dll
  [ 48 b8 10 c6 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/643012be-447e-4580-9bcd-b31ac0200303.dmp 
crashing IP: 0x7ffc0255c984
mismatch: 0x7ffc0255c250 .. 0x7ffc0255c25b (12 bytes) in ntdll.dll
  [ 48 b8 70 c5 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffc0255c850 .. 0x7ffc0255c85b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c6 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffc0255cb50 .. 0x7ffc0255cb5b (12 bytes) in ntdll.dll
  [ 48 b8 10 c6 c4 53 f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/04a1540c-94e5-4e86-a99a-be2dd0200303.dmp 
crashing IP: 0x7ffd5275c984
mismatch: 0x7ffd5275c250 .. 0x7ffd5275c25b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c5 0f 6f f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffd5275c7f0 .. 0x7ffd5275c7f7 (8 bytes) in ntdll.dll
  [ e9 83 39 fe bf cc cc cc ] should be [ 4c 8b d1 b8 3a 00 00 00 ]
mismatch: 0x7ffd5275c850 .. 0x7ffd5275c85b (12 bytes) in ntdll.dll
  [ 48 b8 50 c7 0f 6f f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffd5275c8b0 .. 0x7ffd5275c8b7 (8 bytes) in ntdll.dll
  [ e9 c3 3b fe bf cc cc cc ] should be [ 4c 8b d1 b8 40 00 00 00 ]
mismatch: 0x7ffd5275c9b0 .. 0x7ffd5275c9b7 (8 bytes) in ntdll.dll
  [ e9 43 39 fe bf cc cc cc ] should be [ 4c 8b d1 b8 48 00 00 00 ]
mismatch: 0x7ffd5275cab0 .. 0x7ffd5275cab7 (8 bytes) in ntdll.dll
  [ e9 23 37 fe bf cc cc cc ] should be [ 4c 8b d1 b8 50 00 00 00 ]
mismatch: 0x7ffd5275caf0 .. 0x7ffd5275caf7 (8 bytes) in ntdll.dll
  [ e9 a3 37 fe bf cc cc cc ] should be [ 4c 8b d1 b8 52 00 00 00 ]
mismatch: 0x7ffd5275cb50 .. 0x7ffd5275cb5b (12 bytes) in ntdll.dll
  [ 48 b8 80 c6 0f 6f f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ ./target/release/chkimg --symbol-server https://symbols.mozilla.org/ --symbol-server https://msdl.microsoft.com/download/symbols --symbol-cache /opt/extra/froydnj/tmp/ /opt/build/froydnj/tmp/044d49cd-84ec-40a1-beb6-43beb0200303.dmp 
crashing IP: 0x7ffe6acbc984
mismatch: 0x7ffe6acbc250 .. 0x7ffe6acbc25b (12 bytes) in ntdll.dll
  [ 48 b8 e0 c5 e8 bc f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 0d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffe6acbc710 .. 0x7ffe6acbc71b (12 bytes) in ntdll.dll
  [ 48 b8 10 c7 e8 bc f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 33 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffe6acbc850 .. 0x7ffe6acbc85b (12 bytes) in ntdll.dll
  [ 48 b8 50 c7 e8 bc f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 3d 00 00 00 f6 04 25 08 ]
mismatch: 0x7ffe6acbcb50 .. 0x7ffe6acbcb5b (12 bytes) in ntdll.dll
  [ 48 b8 80 c6 e8 bc f7 7f 00 00 ff e0 ] should be [ 4c 8b d1 b8 55 00 00 00 f6 04 25 08 ]
froydnj@hawkeye:~/src/chkimg$ 

I didn't look what those bytes translate to, but given the similarity across 7 different minidumps (6 installations), I am going to claim that there's some sort of malware on all these machines. (Memory corruption is possible, but given that a number of bytes are very different, rather than just one or two bitflips, and consistency across machines, I think malware is more likely.) Socorro doesn't show any obvious correlations, though.

Comment 8

[Nathan Froyd [:froydnj]]

I am apparently dumb, because what's getting changed in ntdll.dll is our hooking of functions using WindowsDllInterceptor, and the code getting changed is completely unrelated to where the crashes are. Thanks to dmajor for educating me.

So I have no idea what's going on with these content processes, unless we're not going through their proper shutdown sequence due to some nightly-only thing. I don't suppose we're able to correlate these with parent process crashes (or maybe the parent process is shutting down cleanly...), gsvelto?

Comment 9

[Gabriele Svelto [:gsvelto]]

Looking at the IPCShutdownState annotation there's roughly 30% of the crashes which have the annotation set to SendFinishShutdown (sent) so they were just being slow; the process finished shutting down right after we captured the minidump and before we killed the process.

The remaining ones don't have the annotation set at all which indicates that they haven't received the shutdown message yet. This can happen if the process has never been scheduled between when we sent the shutdown IPC message and when we decided to kill it. It's 5 seconds, which is a pretty long time, but it could just indicate slowness.

Which made me remember something important on Windows: since bug 1366356 processes that were not running a foreground tab had their CPU priority demoted. Since we're killing these processes they're certainly not running a foreground tab. A way to speed them up might be to raise their priority again just before we send them the shutdown IPC message. I'll file a bug for that.

Comment 10

[Rares Doghi, Desktop QA]

Considering the "regressionwindow-wanted" tag, I could try to find the regression if some steps to reproduce are provided.
Please NI me if any working STR are obtained. Thanks.

Comment 11

[Julien Cristau [:jcristau]]

This is not a new issue, removed the regression-related flags.

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

Dan - see comment 7

Comment 13

[Daniel Veditz [:dveditz]]

I believe comment 7 is obsolete: comment 8 explains it. I've hidden the comment to avoid future confusion.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

This is one of the top overall Nightly crashes at the moment. Given the resolution of bug 1619676, is there anything else we can do to mitigate this issue?

Comment 15

[Gabriele Svelto [:gsvelto]]

This is just a "content process being slow" issue so I'm dup'ing against bug 1279293 which is where this belongs.