1636333 - Intermittent runner.py | application crashed [@ NtWaitForSingleObject + 0x14]

Status: RESOLVED INVALID

(Core :: DOM: Content Processes, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=301320753&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/B9NvIHJESFeWxQFflk8zaw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-05-07T23:17:50.808Z] 23:17:50 INFO - raptor-webext-desktop Info: starting firefox
[task 2020-05-07T23:17:50.808Z] 23:17:50 INFO - Application command: C:\Users\task_1588884523\build\application\firefox\firefox.exe --allow-downgrade --wait-for-browser -profile c:\users\task_1588884523\appdata\local\temp\tmpje2dec
[task 2020-05-07T23:35:37.790Z] 23:35:37 INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W2mAlWVARquzdH54Becx4g/artifacts/public/build/target.crashreporter-symbols.zip
[task 2020-05-07T23:35:40.111Z] 23:35:40 INFO - mozcrash Copy/paste: C:/Users/task_1588884523/fetches\minidump_stackwalk\minidump_stackwalk.exe c:\users\task_1588884523\appdata\local\temp\tmpje2dec\minidumps\efc83fd9-1e0f-4809-b1cc-a33d8bf9b319.dmp c:\users\task_1588884523\appdata\local\temp\tmpf6zymd
[task 2020-05-07T23:35:50.716Z] 23:35:50 INFO - mozcrash Saved minidump as C:\Users\task_1588884523\build\blobber_upload_dir\efc83fd9-1e0f-4809-b1cc-a33d8bf9b319.dmp
[task 2020-05-07T23:35:50.716Z] 23:35:50 INFO - mozcrash Saved app info as C:\Users\task_1588884523\build\blobber_upload_dir\efc83fd9-1e0f-4809-b1cc-a33d8bf9b319.extra
[task 2020-05-07T23:35:50.717Z] 23:35:50 ERROR - PROCESS-CRASH | runner.py | application crashed [@ NtWaitForSingleObject + 0x14]
[task 2020-05-07T23:35:50.718Z] 23:35:50 INFO - Crash dump filename: c:\users\task_1588884523\appdata\local\temp\tmpje2dec\minidumps\efc83fd9-1e0f-4809-b1cc-a33d8bf9b319.dmp
[task 2020-05-07T23:35:50.718Z] 23:35:50 INFO - Operating system: Windows NT
[task 2020-05-07T23:35:50.718Z] 23:35:50 INFO - 10.0.17134
[task 2020-05-07T23:35:50.719Z] 23:35:50 INFO - CPU: amd64
[task 2020-05-07T23:35:50.719Z] 23:35:50 INFO - family 6 model 142 stepping 9
[task 2020-05-07T23:35:50.719Z] 23:35:50 INFO - 4 CPUs
[task 2020-05-07T23:35:50.719Z] 23:35:50 INFO - GPU: UNKNOWN
[task 2020-05-07T23:35:50.720Z] 23:35:50 INFO - Crash reason: EXCEPTION_BREAKPOINT
[task 2020-05-07T23:35:50.720Z] 23:35:50 INFO - Crash address: 0xe1fdaa24
[task 2020-05-07T23:35:50.720Z] 23:35:50 INFO - Process uptime: 34 seconds
[task 2020-05-07T23:35:50.721Z] 23:35:50 INFO - Thread 0 (crashed)
[task 2020-05-07T23:35:50.721Z] 23:35:50 INFO - 0 ntdll.dll!NtWaitForSingleObject + 0x14
[task 2020-05-07T23:35:50.721Z] 23:35:50 INFO - rax = 0x0000000000000004 rdx = 0x0000000000000001
[task 2020-05-07T23:35:50.721Z] 23:35:50 INFO - rcx = 0x0000000000000714 rbx = 0x0000000000000001
[task 2020-05-07T23:35:50.722Z] 23:35:50 INFO - rsi = 0x0000000000000000 rdi = 0x0000000000000714
[task 2020-05-07T23:35:50.722Z] 23:35:50 INFO - rbp = 0x00000181a4fb9f48 rsp = 0x00000035cabfe428
[task 2020-05-07T23:35:50.722Z] 23:35:50 INFO - r8 = 0x00000181a4f41838 r9 = 0x00000000ffffffff
[task 2020-05-07T23:35:50.722Z] 23:35:50 INFO - r10 = 0x00000181a4f41860 r11 = 0x0000000000000210
[task 2020-05-07T23:35:50.723Z] 23:35:50 INFO - r12 = 0x00007fffb6c7cf82 r13 = 0x00000035cabfe880
[task 2020-05-07T23:35:50.723Z] 23:35:50 INFO - r14 = 0x0000000000000714 r15 = 0x00000035cabfe640
[task 2020-05-07T23:35:50.723Z] 23:35:50 INFO - rip = 0x00007fffe1fdaa24
[task 2020-05-07T23:35:50.723Z] 23:35:50 INFO - Found by: given as instruction pointer in context
[task 2020-05-07T23:35:50.724Z] 23:35:50 INFO - 1 KERNELBASE.dll!LdrpMapAndSnapDependency + 0x152
[task 2020-05-07T23:35:50.724Z] 23:35:50 INFO - rbx = 0x0000000000000001 rbp = 0x00000181a4fb9f48
[task 2020-05-07T23:35:50.724Z] 23:35:50 INFO - rsp = 0x00000035cabfe430 r12 = 0x00007fffb6c7cf82
[task 2020-05-07T23:35:50.724Z] 23:35:50 INFO - r13 = 0x00000035cabfe880 r14 = 0x0000000000000714
[task 2020-05-07T23:35:50.724Z] 23:35:50 INFO - r15 = 0x00000035cabfe640 rip = 0x00007fffdef59252
[task 2020-05-07T23:35:50.725Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.725Z] 23:35:50 INFO - Thread 1
[task 2020-05-07T23:35:50.725Z] 23:35:50 INFO - 0 ntdll.dll!NtRemoveIoCompletion + 0x14
[task 2020-05-07T23:35:50.725Z] 23:35:50 INFO - rax = 0x0000000000000009 rdx = 0x00000035cb3ff7c0
[task 2020-05-07T23:35:50.726Z] 23:35:50 INFO - rcx = 0x000000000000032c rbx = 0x00000035cb3ff7b8
[task 2020-05-07T23:35:50.726Z] 23:35:50 INFO - rsi = 0x0000018194f14700 rdi = 0x00000035cb3ff7b0
[task 2020-05-07T23:35:50.726Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ff708
[task 2020-05-07T23:35:50.726Z] 23:35:50 INFO - r8 = 0x000001819b1880d8 r9 = 0x000001819b1870c0
[task 2020-05-07T23:35:50.727Z] 23:35:50 INFO - r10 = 0x00007fffc295c9b0 r11 = 0x00007fffc2946b00
[task 2020-05-07T23:35:50.727Z] 23:35:50 INFO - r12 = 0x00000035cb3ff7c0 r13 = 0x00007fffe19471f0
[task 2020-05-07T23:35:50.727Z] 23:35:50 INFO - r14 = 0x0000000000000000 r15 = 0x00000035cb3ff7b0
[task 2020-05-07T23:35:50.727Z] 23:35:50 INFO - rip = 0x00007fffe1fdaac4
[task 2020-05-07T23:35:50.728Z] 23:35:50 INFO - Found by: given as instruction pointer in context
[task 2020-05-07T23:35:50.728Z] 23:35:50 INFO - 1 KERNELBASE.dll!LdrResFallbackLangList + 0x232
[task 2020-05-07T23:35:50.728Z] 23:35:50 INFO - rbx = 0x00000035cb3ff7b8 rbp = 0x0000018194f14730
[task 2020-05-07T23:35:50.728Z] 23:35:50 INFO - rsp = 0x00000035cb3ff710 r12 = 0x00000035cb3ff7c0
[task 2020-05-07T23:35:50.729Z] 23:35:50 INFO - r13 = 0x00007fffe19471f0 r14 = 0x0000000000000000
[task 2020-05-07T23:35:50.729Z] 23:35:50 INFO - r15 = 0x00000035cb3ff7b0 rip = 0x00007fffdef736b2
[task 2020-05-07T23:35:50.729Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.729Z] 23:35:50 INFO - 2 0x18194f0a480
[task 2020-05-07T23:35:50.730Z] 23:35:50 INFO - rbx = 0x00000035cb3ff7b8 rbp = 0x0000018194f14730
[task 2020-05-07T23:35:50.730Z] 23:35:50 INFO - rsp = 0x00000035cb3ff7f0 r12 = 0x00000035cb3ff7c0
[task 2020-05-07T23:35:50.730Z] 23:35:50 INFO - r13 = 0x00007fffe19471f0 r14 = 0x0000000000000000
[task 2020-05-07T23:35:50.730Z] 23:35:50 INFO - r15 = 0x00000035cb3ff7b0 rip = 0x0000018194f0a480
[task 2020-05-07T23:35:50.731Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.731Z] 23:35:50 INFO - 3 xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate*) [message_pump_win.h:f9356df325cd5df64ef952636b788db27fbfd19f : 79 + 0x4f]
[task 2020-05-07T23:35:50.731Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ff830
[task 2020-05-07T23:35:50.731Z] 23:35:50 INFO - rip = 0x00007fffb1d5af9f
[task 2020-05-07T23:35:50.732Z] 23:35:50 INFO - Found by: stack scanning
[task 2020-05-07T23:35:50.732Z] 23:35:50 INFO - 4 xul.dll!MessageLoop::RunHandler() [message_loop.cc:f9356df325cd5df64ef952636b788db27fbfd19f : 308 + 0xf]
[task 2020-05-07T23:35:50.732Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ff890
[task 2020-05-07T23:35:50.732Z] 23:35:50 INFO - rip = 0x00007fffb2bba5e8
[task 2020-05-07T23:35:50.733Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.733Z] 23:35:50 INFO - 5 xul.dll!MessageLoop::Run() [message_loop.cc:f9356df325cd5df64ef952636b788db27fbfd19f : 290 + 0x5]
[task 2020-05-07T23:35:50.733Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ff8e0
[task 2020-05-07T23:35:50.733Z] 23:35:50 INFO - rip = 0x00007fffb1d5af11
[task 2020-05-07T23:35:50.734Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.734Z] 23:35:50 INFO - 6 xul.dll!base::Thread::ThreadMain() [thread.cc:f9356df325cd5df64ef952636b788db27fbfd19f : 192 + 0x8]
[task 2020-05-07T23:35:50.734Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ff930
[task 2020-05-07T23:35:50.734Z] 23:35:50 INFO - rip = 0x00007fffb1d593f6
[task 2020-05-07T23:35:50.735Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.735Z] 23:35:50 INFO - 7 xul.dll!`anonymous namespace'::ThreadFunc(void*) [platform_thread_win.cc:f9356df325cd5df64ef952636b788db27fbfd19f : 19 + 0x6]
[task 2020-05-07T23:35:50.735Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ffaf0
[task 2020-05-07T23:35:50.735Z] 23:35:50 INFO - rip = 0x00007fffb2bb354a
[task 2020-05-07T23:35:50.736Z] 23:35:50 INFO - Found by: call frame info
[task 2020-05-07T23:35:50.736Z] 23:35:50 INFO - 8 kernel32.dll!RtlpReAllocateHeap + 0xe4
[task 2020-05-07T23:35:50.736Z] 23:35:50 INFO - rbp = 0x0000018194f14730 rsp = 0x00000035cb3ffb20
[task 2020-05-07T23:35:50.736Z] 23:35:50 INFO - rip = 0x00007fffe1944034
[task 2020-05-07T23:35:50.737Z] 23:35:50 INFO - Found by: call frame info

Comment 1

[Intermittent Failures Robot]

1 failures in 5276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 2

[Firefox Bug Husbandry Bot]

Because this bug's Severity is normal and has not been changed, and this bug's priority is -- (none,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is a crash of Firefox and not related to Raptor at all. The first stack frame containing a code from our code base is for message_loop.cc. So moving the bug to IPC.

Comment 4

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

This isn't a crash in NtWaitForSingleObject though — it's the result of ContentParent::KillHard, typically a watchdog timeout during shutdown, and the thread in the allegedly hanging content process that was (arbitrarily?) treated as the “current” thread was the IPC I/O thread, which was idle.

Comment 5

[Intermittent Failures Robot]

1 failures in 5488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 6

[Chris Peterson [:cpeterson]]

P3

block ShutdownKill meta bug 1279293

Comment 7

[Intermittent Failures Robot]

1 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 8

[Tim Spurway [:tspurway]]

S1 or S2 bugs need an assignee - could you find someone for this bug?

Comment 9

[Andrew McCreight [:mccr8]]

This is an intermittent test failure that has only happened twice in the last month. Is S2 the right rating for this? If it happens it is bad, but we don't have any evidence here that it happens often to any user.

Comment 10

[Chris Peterson [:cpeterson]]

(In reply to Andrew McCreight [:mccr8] from comment #9)

This is an intermittent test failure that has only happened twice in the last month. Is S2 the right rating for this? If it happens it is bad, but we don't have any evidence here that it happens often to any user.

I'll drop this bug severity to S3 and priority to P5 because this is a very low volume test crash that hasn't affected actual users.

Comment 11

[Intermittent Failures Robot]

1 failures in 5458 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 4949 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-64-ref-hw-2017: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 5265 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1636333&startday=2020-07-06&endday=2020-07-12&tree=all