Closed Bug 1496825 Opened Last year Closed Last year

Intermittent Win10 QR test_sanityEventUtils.html PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6]

Categories

(Core :: Graphics: WebRender, defect, P5, critical)

defect

Tracking

()

RESOLVED FIXED
mozilla65
Tracking Status
firefox-esr60 --- unaffected
firefox63 --- unaffected
firefox64 --- disabled
firefox65 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: sotaro)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(1 file)

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=203695484&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/WjNlTOxTSj-YhSf9aL9mXg/runs/0/artifacts/public/logs/live_backing.log

17:06:37     INFO - TEST-START | testing/mochitest/tests/Harness_sanity/test_sanityEventUtils.html
17:06:38     INFO - GECKO(832) | MEMORY STAT | vsize 1487MB | vsizeMaxContiguous 130417311MB | residentFast 90MB | heapAllocated 22MB
17:06:38     INFO - GECKO(832) | ###!!! [Parent][MessageChannel] Error: (msgtype=0x190091,name=PBrowser::Msg_UpdateNativeWindowHandle) Channel error: cannot send/recv
17:06:38     INFO - GECKO(832) | ###!!! [Parent][MessageChannel] Error: (msgtype=0x190084,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
17:06:38     INFO - GECKO(832) | A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
17:06:38     INFO - GECKO(832) | [Parent 7348, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:38     INFO - GECKO(832) | [Parent 7348, Gecko_IOThread] WARNING: file z:/build/build/src/ipc/chromium/src/base/process_util_win.cc, line 188
17:06:38     INFO - GECKO(832) | [Child 9168, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:38     INFO - GECKO(832) | [Child 9168, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:39     INFO - GECKO(832) | [Parent 7348, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:39     INFO - GECKO(832) | [Child 5848, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:39     INFO - GECKO(832) | 1538759199278	Marionette	DEBUG	Received observer notification xpcom-will-shutdown
17:06:39     INFO - GECKO(832) | 1538759199278	Marionette	INFO	Stopped listening on port 2828
17:06:39     INFO - GECKO(832) | 1538759199278	Marionette	DEBUG	Remote service is inactive
17:06:39     INFO - GECKO(832) | [GPU 11296, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
17:06:39     INFO - GECKO(832) | ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
17:06:39     INFO - TEST-INFO | Main app process: exit 0
17:06:39     INFO - runtests.py | Application ran for: 0:00:12.267000
17:06:39     INFO - zombiecheck | Reading PID log: c:\users\task_1538758094\appdata\local\temp\tmpj8tkutpidlog
17:06:39     INFO - ==> process 7348 launched child process 11296 ("Z:\task_1538758094\build\application\firefox\firefox.exe" -contentproc --channel="7348.0.1842997950\267269970" -parentBuildID 20181005073554 -greomni "Z:\task_1538758094\build\application\firefox\omni.ja" -appomni "Z:\task_1538758094\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1538758094\build\application\firefox\browser" - "C:\Users\task_1538758094\AppData\LocalLow\Mozilla\Temp-{7078e3c1-7f60-46b5-9a73-6d0e79d601de}" 7348 "\\.\pipe\gecko-crash-server-pipe.7348" 2200 gpu)
17:06:39     INFO - ==> process 7348 launched child process 5848 ("Z:\task_1538758094\build\application\firefox\firefox.exe" -contentproc --channel="7348.6.2122266297\1815681225" -childID 1 -isForBrowser -prefsHandle 2644 -prefMapHandle 2576 -prefsLen 1 -prefMapSize 187778 -schedulerPrefs 0001,2 -parentBuildID 20181005073554 -greomni "Z:\task_1538758094\build\application\firefox\omni.ja" -appomni "Z:\task_1538758094\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1538758094\build\application\firefox\browser" - 7348 "\\.\pipe\gecko-crash-server-pipe.7348" 2696 tab)
17:06:39     INFO - ==> process 7348 launched child process 9168 ("Z:\task_1538758094\build\application\firefox\firefox.exe" -contentproc --channel="7348.13.2033237570\1716550675" -childID 2 -isForBrowser -prefsHandle 2868 -prefMapHandle 2864 -prefsLen 1 -prefMapSize 187778 -schedulerPrefs 0001,2 -parentBuildID 20181005073554 -greomni "Z:\task_1538758094\build\application\firefox\omni.ja" -appomni "Z:\task_1538758094\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1538758094\build\application\firefox\browser" - 7348 "\\.\pipe\gecko-crash-server-pipe.7348" 2600 tab)
17:06:39     INFO - ==> process 7348 launched child process 12000 ("Z:\task_1538758094\build\application\firefox\firefox.exe" -contentproc --channel="7348.20.327199604\922706441" -childID 3 -isForBrowser -prefsHandle 1720 -prefMapHandle 1724 -prefsLen 41 -prefMapSize 187778 -schedulerPrefs 0001,2 -parentBuildID 20181005073554 -greomni "Z:\task_1538758094\build\application\firefox\omni.ja" -appomni "Z:\task_1538758094\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1538758094\build\application\firefox\browser" - 7348 "\\.\pipe\gecko-crash-server-pipe.7348" 1888 tab)
17:06:39     INFO - zombiecheck | Checking for orphan process with PID: 11296
17:06:39     INFO - zombiecheck | Checking for orphan process with PID: 5848
17:06:39     INFO - zombiecheck | Checking for orphan process with PID: 9168
17:06:39     INFO - zombiecheck | Checking for orphan process with PID: 12000
17:06:39     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/H35zC4okSMeTL1kVbCNe-Q/artifacts/public/build/target.crashreporter-symbols.zip
17:06:45     INFO - mozcrash Copy/paste: Z:\task_1538758094\build\win32-minidump_stackwalk.exe c:\users\task_1538758094\appdata\local\temp\tmpeopiuw.mozrunner\minidumps\cf6d1ef9-80ed-48cb-8018-42beb8fa408f.dmp c:\users\task_1538758094\appdata\local\temp\tmpew_19m
17:06:48     INFO - mozcrash Saved minidump as Z:\task_1538758094\build\blobber_upload_dir\cf6d1ef9-80ed-48cb-8018-42beb8fa408f.dmp
17:06:48     INFO - mozcrash Saved app info as Z:\task_1538758094\build\blobber_upload_dir\cf6d1ef9-80ed-48cb-8018-42beb8fa408f.extra
17:06:48     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6]
17:06:48     INFO - Crash dump filename: c:\users\task_1538758094\appdata\local\temp\tmpeopiuw.mozrunner\minidumps\cf6d1ef9-80ed-48cb-8018-42beb8fa408f.dmp
17:06:48     INFO - Operating system: Windows NT
17:06:48     INFO -                   10.0.15063 
17:06:48     INFO - CPU: amd64
17:06:48     INFO -      family 6 model 45 stepping 7
17:06:48     INFO -      8 CPUs
17:06:48     INFO - 
17:06:48     INFO - GPU: UNKNOWN
17:06:48     INFO - 
17:06:48     INFO - Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
17:06:48     INFO - Crash address: 0x60
17:06:48     INFO - Assertion: Unknown assertion type 0x00000000
17:06:48     INFO - Process uptime: 9 seconds
17:06:48     INFO - 
17:06:48     INFO - Thread 0 (crashed)
17:06:48     INFO -  0  xul.dll + 0xbc90c6
17:06:48     INFO -     rax = 0x0000000000000000   rdx = 0x0000000000000000
17:06:48     INFO -     rcx = 0x00000143070a3020   rbx = 0x00000143070a3020
17:06:48     INFO -     rsi = 0x000001430f4a30a0   rdi = 0x000001430f4a30c0
17:06:48     INFO -     rbp = 0x0000000000000001   rsp = 0x00000004457fe410
17:06:48     INFO -      r8 = 0x0000000000000000    r9 = 0x0000000000000000
17:06:48     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
17:06:48     INFO -     r12 = 0x00007ff96a4ef990   r13 = 0x000001430da7d000
17:06:48     INFO -     r14 = 0x000001430da67a80   r15 = 0x000001430da14800
17:06:48     INFO -     rip = 0x00007ff9664090c6
17:06:48     INFO -     Found by: given as instruction pointer in context
17:06:48     INFO -  1  mozglue.dll!static bool malloc_init() [mozjemalloc.cpp:0052f643487612ccd8993a8687b1a3b1cbede32d : 1283 + 0xd]
17:06:48     INFO -     rsp = 0x00000004457fe6d0   rip = 0x00007ff98ecaf960
17:06:48     INFO -     Found by: stack scanning
17:06:48     INFO -  2  0x1430f1b8a00
17:06:48     INFO -     rsp = 0x00000004457fe770   rip = 0x000001430f1b8a00
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  3  mozglue.dll!Allocator<MozJemallocBase>::malloc(unsigned __int64) [malloc_decls.h:0052f643487612ccd8993a8687b1a3b1cbede32d : 37 + 0x18]
17:06:48     INFO -     rsp = 0x00000004457fe7c0   rip = 0x00007ff98ecae4cd
17:06:48     INFO -     Found by: stack scanning
17:06:48     INFO -  4  0x2c7896acc09a
17:06:48     INFO -     rsp = 0x00000004457fe800   rip = 0x00002c7896acc09a
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  5  ntdll.dll!RtlQueryPerformanceCounter + 0x53
17:06:48     INFO -     rsp = 0x00000004457fe930   rip = 0x00007ff99a7cc7d3
17:06:48     INFO -     Found by: stack scanning
17:06:48     INFO -  6  mozglue.dll!mozilla::TimeStamp::Now(bool) [TimeStamp_windows.cpp:0052f643487612ccd8993a8687b1a3b1cbede32d : 541 + 0x2c]
17:06:48     INFO -     rsp = 0x00000004457fe960   rip = 0x00007ff98ecb6792
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  7  xul.dll + 0x23a45f6
17:06:48     INFO -     rsp = 0x00000004457fe9c0   rip = 0x00007ff967be45f6
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  8  ucrtbase.dll!RtlFreeAnsiString + 0x2d
17:06:48     INFO -     rsp = 0x00000004457feae0   rip = 0x00007ff996e2c48d
17:06:48     INFO -     Found by: stack scanning
17:06:48     INFO - 
17:06:48     INFO - Thread 1
17:06:48     INFO -  0  ntdll.dll!NtRemoveIoCompletion + 0x14
17:06:48     INFO -     rax = 0x0000014307082930   rdx = 0x00000143070790c8
17:06:48     INFO -     rcx = 0x0000014307073400   rbx = 0x00000004459ffa18
17:06:48     INFO -     rsi = 0x0000014301a0e8e0   rdi = 0x00000004459ffa10
17:06:48     INFO -     rbp = 0x0000000000000000   rsp = 0x00000004459ff968
17:06:48     INFO -      r8 = 0x0000000000000008    r9 = 0x0000000000000000
17:06:48     INFO -     r10 = 0x00000143070780c0   r11 = 0x000001430f4676c0
17:06:48     INFO -     r12 = 0x0000000000000000   r13 = 0x0000000000000000
17:06:48     INFO -     r14 = 0x00000004459ffa70   r15 = 0x00000004459ffa78
17:06:48     INFO -     rip = 0x00007ff99a8154c4
17:06:48     INFO -     Found by: given as instruction pointer in context
17:06:48     INFO -  1  KERNELBASE.dll!LdrResGetRCConfig + 0x40f
17:06:48     INFO -     rbx = 0x00000004459ffa18   rbp = 0x0000000000000000
17:06:48     INFO -     rsp = 0x00000004459ff970   r12 = 0x0000000000000000
17:06:48     INFO -     r13 = 0x0000000000000000   r14 = 0x00000004459ffa70
17:06:48     INFO -     r15 = 0x00000004459ffa78   rip = 0x00007ff997079aff
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  2  0x16
17:06:48     INFO -     rbx = 0x00000004459ffa18   rbp = 0x0000000000000000
17:06:48     INFO -     rsp = 0x00000004459ffa80   r12 = 0x0000000000000000
17:06:48     INFO -     r13 = 0x0000000000000000   r14 = 0x00000004459ffa70
17:06:48     INFO -     r15 = 0x00000004459ffa78   rip = 0x0000000000000016
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  3  KERNELBASE.dll!ResCKeOpenRuntimeView + 0x114
17:06:48     INFO -     rsp = 0x00000004459ffb40   rip = 0x00007ff99707ab9c
17:06:48     INFO -     Found by: stack scanning
17:06:48     INFO -  4  0x7ff965840000
17:06:48     INFO -     rsp = 0x00000004459ffdb0   rip = 0x00007ff965840000
17:06:48     INFO -     Found by: call frame info
17:06:48     INFO -  5  ntdll.dll!RtlUserThreadStart + 0x21
17:06:48     INFO -     rsp = 0x00000004459ffe00   rip = 0x00007ff99a7e0d51
17:06:49     INFO -     Found by: stack scanning
Based on retriggers this started happening when bug 1496003 landed:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=success,testfailed,busted,exception&fromchange=e8aec0dccbb8213dd292d7331f01307969cdff9a&searchStr=windows,10,x64,quantumrender,opt,mochitests,with,e10s,test-windows10-64-qr%2Fopt-mochitest-e10s-5,m-e10s(5)&selectedJob=203695484

This is rather frequent and only occurs on Windows 10 x64 QuantumRender opt, Sotaro can you please take a look? 

I've checked all occurrences so far, all fail in testing/mochitest/tests/Harness_sanity/test_sanityEventUtils.html
Component: Untriaged → Mochitest
Product: Firefox → Testing
Summary: Intermittent PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6] → Intermittent Win10 QR PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6]
Flags: needinfo?(sotaro.ikeda.g)
Ok, I'll look into it. The crash stack seems not have enough information.
Assignee: nobody → sotaro.ikeda.g
Flags: needinfo?(sotaro.ikeda.g)
Crash Signature: [@ xul.dll + 0xbc90c6] → [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6]
Crash Signature: [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] → [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] [@ xul.dll + 0xbd12c6]
The intermittent became less often. It might be related to recent share shader related works.
Summary: Intermittent Win10 QR PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6] → Intermittent Win10 QR test_sanityEventUtils.html PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbc90c6]
Crash Signature: [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] [@ xul.dll + 0xbd12c6] → [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] [@ xul.dll + 0xbd12c6] [@ xul.dll + 0xbd67a6]
https://treeherder.mozilla.org/logviewer.html#?job_id=206740930&repo=autoland&lineNumber=4086

02:05:26     INFO - TEST-START | testing/mochitest/tests/Harness_sanity/test_sanityEventUtils.html
02:05:27     INFO - GECKO(10848) | MEMORY STAT | vsize 1496MB | vsizeMaxContiguous 130544656MB | residentFast 90MB | heapAllocated 22MB
02:05:27     INFO - GECKO(10848) | ###!!! [Parent][MessageChannel] Error: (msgtype=0x190095,name=PBrowser::Msg_UpdateNativeWindowHandle) Channel error: cannot send/recv
02:05:27     INFO - GECKO(10848) | ###!!! [Parent][MessageChannel] Error: (msgtype=0x190084,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
02:05:27     INFO - GECKO(10848) | A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
02:05:27     INFO - GECKO(10848) | [Parent 6888, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:27     INFO - GECKO(10848) | [Parent 6888, Gecko_IOThread] WARNING: file z:/build/build/src/ipc/chromium/src/base/process_util_win.cc, line 188
02:05:27     INFO - GECKO(10848) | [Child 10664, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:27     INFO - GECKO(10848) | [Child 10664, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:27     INFO - GECKO(10848) | [Parent 6888, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:27     INFO - GECKO(10848) | [Child 12012, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:27     INFO - GECKO(10848) | [Child 12012, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:28     INFO - GECKO(10848) | 1540001128241	Marionette	DEBUG	Received observer notification xpcom-will-shutdown
02:05:28     INFO - GECKO(10848) | 1540001128241	Marionette	INFO	Stopped listening on port 2828
02:05:28     INFO - GECKO(10848) | 1540001128242	Marionette	DEBUG	Remote service is inactive
02:05:28     INFO - GECKO(10848) | [GPU 11272, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
02:05:28     INFO - GECKO(10848) | ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
02:05:28     INFO - TEST-INFO | Main app process: exit 0
02:05:28     INFO - runtests.py | Application ran for: 0:00:11.563000
02:05:28     INFO - zombiecheck | Reading PID log: c:\users\task_1539999711\appdata\local\temp\tmppfy_lhpidlog
02:05:28     INFO - ==> process 6888 launched child process 11272 ("Z:\task_1539999711\build\application\firefox\firefox.exe" -contentproc --channel="6888.0.693184654\556778016" -parentBuildID 20181019221014 -greomni "Z:\task_1539999711\build\application\firefox\omni.ja" -appomni "Z:\task_1539999711\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1539999711\build\application\firefox\browser" - "C:\Users\task_1539999711\AppData\LocalLow\Mozilla\Temp-{84dc3884-2f47-40a9-b9e6-7d632a29c2f4}" 6888 "\\.\pipe\gecko-crash-server-pipe.6888" 2516 gpu)
02:05:28     INFO - ==> process 6888 launched child process 12012 ("Z:\task_1539999711\build\application\firefox\firefox.exe" -contentproc --channel="6888.6.45132762\264272306" -childID 1 -isForBrowser -prefsHandle 2684 -prefMapHandle 2680 -prefsLen 1 -prefMapSize 188141 -schedulerPrefs 0001,2 -parentBuildID 20181019221014 -greomni "Z:\task_1539999711\build\application\firefox\omni.ja" -appomni "Z:\task_1539999711\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1539999711\build\application\firefox\browser" - 6888 "\\.\pipe\gecko-crash-server-pipe.6888" 2928 tab)
02:05:28     INFO - ==> process 6888 launched child process 10664 ("Z:\task_1539999711\build\application\firefox\firefox.exe" -contentproc --channel="6888.13.656796588\905536384" -childID 2 -isForBrowser -prefsHandle 2884 -prefMapHandle 2868 -prefsLen 1 -prefMapSize 188141 -schedulerPrefs 0001,2 -parentBuildID 20181019221014 -greomni "Z:\task_1539999711\build\application\firefox\omni.ja" -appomni "Z:\task_1539999711\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1539999711\build\application\firefox\browser" - 6888 "\\.\pipe\gecko-crash-server-pipe.6888" 2680 tab)
02:05:28     INFO - ==> process 6888 launched child process 1540 ("Z:\task_1539999711\build\application\firefox\firefox.exe" -contentproc --channel="6888.20.580689905\1174932315" -childID 3 -isForBrowser -prefsHandle 3088 -prefMapHandle 3084 -prefsLen 41 -prefMapSize 188141 -schedulerPrefs 0001,2 -parentBuildID 20181019221014 -greomni "Z:\task_1539999711\build\application\firefox\omni.ja" -appomni "Z:\task_1539999711\build\application\firefox\browser\omni.ja" -appdir "Z:\task_1539999711\build\application\firefox\browser" - 6888 "\\.\pipe\gecko-crash-server-pipe.6888" 2768 tab)
02:05:28     INFO - zombiecheck | Checking for orphan process with PID: 11272
02:05:28     INFO - zombiecheck | Checking for orphan process with PID: 10664
02:05:28     INFO - zombiecheck | Checking for orphan process with PID: 12012
02:05:28     INFO - zombiecheck | Checking for orphan process with PID: 1540
02:05:28     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/OX_SRb1vStGy45lIQvNuJA/artifacts/public/build/target.crashreporter-symbols.zip
02:05:37     INFO - mozcrash Copy/paste: Z:\task_1539999711\build\win32-minidump_stackwalk.exe c:\users\task_1539999711\appdata\local\temp\tmpaljvfh.mozrunner\minidumps\5d7f4ade-f22b-47b8-ab86-e9f9631000a2.dmp c:\users\task_1539999711\appdata\local\temp\tmp65nxsj
02:05:41     INFO - mozcrash Saved minidump as Z:\task_1539999711\build\blobber_upload_dir\5d7f4ade-f22b-47b8-ab86-e9f9631000a2.dmp
02:05:41     INFO - mozcrash Saved app info as Z:\task_1539999711\build\blobber_upload_dir\5d7f4ade-f22b-47b8-ab86-e9f9631000a2.extra
02:05:41     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ xul.dll + 0xbd67a6]
02:05:41     INFO - Crash dump filename: c:\users\task_1539999711\appdata\local\temp\tmpaljvfh.mozrunner\minidumps\5d7f4ade-f22b-47b8-ab86-e9f9631000a2.dmp
02:05:41     INFO - Operating system: Windows NT
02:05:41     INFO -                   10.0.15063 
02:05:41     INFO - CPU: amd64
02:05:41     INFO -      family 6 model 62 stepping 4
02:05:41     INFO -      8 CPUs
02:05:41     INFO - 
02:05:41     INFO - GPU: UNKNOWN
02:05:41     INFO - 
02:05:41     INFO - Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
02:05:41     INFO - Crash address: 0x60
02:05:41     INFO - Assertion: Unknown assertion type 0x00000000
02:05:41     INFO - Process uptime: 8 seconds
02:05:41     INFO - 
02:05:41     INFO - Thread 0 (crashed)
02:05:41     INFO -  0  xul.dll + 0xbd67a6
02:05:41     INFO -     rax = 0x0000000000000000   rdx = 0x0000000000000000
02:05:41     INFO -     rcx = 0x0000019e39fab020   rbx = 0x0000019e39fab020
02:05:41     INFO -     rsi = 0x0000019e4250d1f0   rdi = 0x0000019e4250d210
02:05:41     INFO -     rbp = 0x0000000000000001   rsp = 0x00000003f47fe330
02:05:41     INFO -      r8 = 0x0000000000000000    r9 = 0x0000000000000000
02:05:41     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
02:05:41     INFO -     r12 = 0x00007ff863f8c188   r13 = 0x0000019e40687000
02:05:41     INFO -     r14 = 0x0000019e40674ac0   r15 = 0x0000019e40619000
02:05:41     INFO -     rip = 0x00007ff85fe967a6
02:05:41     INFO -     Found by: given as instruction pointer in context
02:05:41     INFO -  1  mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:a14911472fe733a76c7703a29e35440f8f1c75cd : 4170 + 0x32]
02:05:41     INFO -     rsp = 0x00000003f47fe590   rip = 0x00007ff88239fc00
02:05:41     INFO -     Found by: stack scanning
02:05:41     INFO -  2  mozglue.dll!arena_t::MallocSmall(unsigned __int64,bool) [mozjemalloc.cpp:a14911472fe733a76c7703a29e35440f8f1c75cd : 2983 + 0x9]
02:05:41     INFO -     rsp = 0x00000003f47fe5c0   rip = 0x00007ff88239ee31
02:05:41     INFO -     Found by: stack scanning
02:05:41     INFO -  3  mozglue.dll!BaseAllocator::malloc(unsigned __int64) [mozjemalloc.cpp:a14911472fe733a76c7703a29e35440f8f1c75cd : 4163 + 0x23]
02:05:41     INFO -     rsp = 0x00000003f47fe630   rip = 0x00007ff88239fb46
02:05:41     INFO -     Found by: stack scanning
02:05:41     INFO -  4  mozglue.dll!Allocator<MozJemallocBase>::malloc(unsigned __int64) [malloc_decls.h:a14911472fe733a76c7703a29e35440f8f1c75cd : 37 + 0x18]
02:05:41     INFO -     rsp = 0x00000003f47fe680   rip = 0x00007ff88239b53d
02:05:41     INFO -     Found by: stack scanning
Crash Signature: [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] [@ xul.dll + 0xbd12c6] [@ xul.dll + 0xbd67a6] → [@ xul.dll + 0xbc90c6] [@ xul.dll + 0xbcccb6] [@ xul.dll + 0xbd12c6] [@ xul.dll + 0xbd67a6] [@ xul.dll + 0xbd67b6]
Component: Mochitest → Memory Allocator
Product: Testing → Core
The malloc symbols in the stack trace are given from stack scanning which is unreliable. They are probably a red herring (malloc doesn't call back into libxul anyways). It's weird that the top frame isn't symbolicated.

David, can you make something out of the minidump?
Flags: needinfo?(dmajor)
We don't generate fullsymbols on autoland as of bug 1466537. :(
Flags: needinfo?(dmajor)
Ah, I missed that, thanks.

`lm->AsShadowForwarder()` is null at https://searchfox.org/mozilla-central/rev/fcfb479e6ff63aea017d063faa17877ff750b4e5/gfx/layers/apz/util/APZCCallbackHelper.cpp#755
Flags: needinfo?(dmajor)
Separately from the problem itself, it would be good to know why the top of the stack is not being symbolicated.
Component: Memory Allocator → Panning and Zooming
Per comment 1 this is a regression from bug 1496003 and specific to WR. If the crash is that `lm->AsShadowForwarder()` is null (per comment 17) that's pretty weird, since `lm` must be non-null, not a WebRenderLayerManager, and not a ClientLayerManager. Maybe we're falling back to BasicLayerManager somehow? There are other WebRenderBridgeChild error messages, e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=206623508&repo=mozilla-central&lineNumber=3953

Sotaro, any ideas what's going on here?
Component: Panning and Zooming → Graphics: WebRender
Flags: needinfo?(sotaro.ikeda.g)
> Maybe we're falling back to BasicLayerManager somehow?

Yeah, looks like it:

0:000> dqs @rbx
000002aa`d09aa020  00007ffc`d26c7e80 xul!mozilla::layers::BasicLayerManager::`vftable'
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #19)
> Per comment 1 this is a regression from bug 1496003 and specific to WR. If
> the crash is that `lm->AsShadowForwarder()` is null (per comment 17) that's
> pretty weird, since `lm` must be non-null, not a WebRenderLayerManager, and
> not a ClientLayerManager. Maybe we're falling back to BasicLayerManager
> somehow? There are other WebRenderBridgeChild error messages, e.g.
> https://treeherder.mozilla.org/logviewer.html#?job_id=206623508&repo=mozilla-
> central&lineNumber=3953
> 
> Sotaro, any ideas what's going on here?

WebRenderBridgeChild error messages seems to related to gfx/SanityTest failure. When the test failed, it triggers device reset to update LayerManager. If the test failure happened just before creating WebRenderBridgeChild, it caused the WebRenderBridgeChild creation failure. But it is not necessary when WebRender is used. The device reset was suppressed in Bug 1499002. It was checked-in yesterday.
Flags: needinfo?(sotaro.ikeda.g)
Depends on: 1499002
(In reply to David Major [:dmajor] from comment #20)
> > Maybe we're falling back to BasicLayerManager somehow?
> 
> Yeah, looks like it:
> 
> 0:000> dqs @rbx
> 000002aa`d09aa020  00007ffc`d26c7e80
> xul!mozilla::layers::BasicLayerManager::`vftable'

Great! Thanks!!

gfx/SanityTest failure seems to happen often because WebRender sometimes causes longer load time than normal Compositor because of shader compiling. The device reset was suppressed in Bug 1499002. 

The gfx/SanityTest failure could happen also for ClientLayerManager. In this case, it also fallback to BasicLayerManager. Then it seems better to add check if lm->AsShadowForwarder() is not null.
Attachment #9019561 - Flags: review?(kats)
Pushed by sikeda@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/109ff0694f9a
Add ShadowLayerForwarder nullptr check to SendLayersDependentApzcTargetConfirmation() r=kats
https://hg.mozilla.org/mozilla-central/rev/109ff0694f9a
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Blocks: 1492397
You need to log in before you can comment on or make changes to this bug.