1518895 - Enable crash reporting for the VR process

Status: RESOLVED FIXED

(Core :: WebVR, enhancement, P3)

Description

[Nathan Froyd [:froydnj]]

This functionality is not enabled, and really should be, since we are shipping the VR process in Nightly. Discovered when handling bug 1514044, and may slightly easier once that bug lands.

Comment 1

[Sotaro Ikeda [:sotaro]]

:daoshengmu, can you work for the bug? Since you implemented VR process.

Comment 2

[Daosheng Mu[:daoshengmu]]

Sure. I will take this bug. Thanks.

Comment 3

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1518895 - Add crash report for VR process.

Comment 4

[Daosheng Mu[:daoshengmu]]

(In reply to Daosheng Mu[:daoshengmu] from comment #3)

Created attachment 9037002 [details]
Bug 1518895 - Add crash report for VR process.

:gsvelto, I tried to use ./mach test toolkit/components/crashes/tests/xpcshell/test_crash_manager.js on Windows with the current m-c. but I got a fail as below, at https://searchfox.org/mozilla-central/rev/c21d6620d384dfb13ede6054015da05a6353b899/toolkit/components/crashes/tests/xpcshell/test_crash_manager.js#262.

0:16.16 INFO exiting test
0:16.16 ERROR Unexpected exception NS_ERROR_ABORT:
_abort_failed_test@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:741:9
do_report_result@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:848:5
Assert<@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:54:5
proto.report@resource://testing-common/Assert.jsm:214:5
proto.ok@resource://testing-common/Assert.jsm:234:5
test_main_crash_event_file@c:/Projects/mozilla/gecko_build/_tests/xpcshell/toolkit/components/crashes/tests/xpcshell/test_crash_manager.js:262:3
asyncrun_next_test/_run_next_test/<@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:1436:22
async_run_next_test@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:1436:10
run@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:687:9
_do_main@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:224:3
_execute_test@c:\Projects\mozilla\gecko-dev\testing\xpcshell\head.js:528:5
@-e:1:1

However, it is good on Mac OS. Also, I would like to know how to verify if this crash report works. I was trying to call MOZ_CRASH() at VR process or GPU process and put breakpoints in CrashReporterHost.cpp at the parent process. But I didn't get any events about the report. Even typing about::crashes, I still didn't see any reports. I notice I can get VRChild::ActorDestroy() for generating the crash report, but in TakeCrashedChildMinidump(), it returns false. It seems like I didn't enable the CrashReporter::TakeMinidumpForChild(). Could you tell me have to generate this mini dump?

Thanks,

Comment 5

[Gabriele Svelto [:gsvelto]]

I gave your patch a quick look and everything seems fine, I will test it on my machine tomorrow to figure out what's going on. In the meantime if you need to test crash reporting on Windows the best way is to use the crashfirefox-intentionally [1] tool. You can pass it a PID to the process you need to crash and it will crash it for you. If you started your local build with ./mach run --enable-crash-reporter it should work fine. Leaving the NI? until I have time to test the patch myself.

[1] https://github.com/bsmedberg/crashfirefox-intentionally

Comment 6

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1518895 - Part 1: Add crash report test for VR process.

Comment 7

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1518895 - Part 2: Crash report support in VR process.

Comment 8

[Gabriele Svelto [:gsvelto]]

Just tested the patch and the tests run fine in my Windows build.

Comment 9

[Daosheng Mu[:daoshengmu]]

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

Just tested the patch and the tests run fine in my Windows build.

How do you run the test? and be interested in the configs in your .mozconfig.

I run ./mach test toolkit/components/crashes/tests/xpcshell/test_crash_manager.js still gets fail after applying my patches.

I am using Windows Process Explorer to kill processes.

After calling ./mach run --enable-crash-reporter and kill GPU or VR process, it enables CrashReporter::TakeMinidumpForChild() but still has no data at ChildProcessData* pd = pidToMinidump->GetEntry(childPid).

Reproduce steps:

1. ./mach run --enable-crash-reporter on Windows
2. Go to https://webvr.info/samples/04-simple-mirroring.html, you will see a VR process is launched.
3. Using MSVC debugger or gdb to attach the parent process.
4. Kill VR process in Process Explorer.
5. The breakpoint will be triggered at CrashReporter::TakeMinidumpForChild() in the parent process.

Comment 10

[Daosheng Mu[:daoshengmu]]

(In reply to Daosheng Mu[:daoshengmu] from comment #9)

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

Just tested the patch and the tests run fine in my Windows build.

How do you run the test? and be interested in the configs in your .mozconfig.

I run ./mach test toolkit/components/crashes/tests/xpcshell/test_crash_manager.js still gets fail after applying my patches.

I am using Windows Process Explorer to kill processes.

After calling ./mach run --enable-crash-reporter and kill GPU or VR process, it enables CrashReporter::TakeMinidumpForChild() but still has no data at ChildProcessData* pd = pidToMinidump->GetEntry(childPid).

Reproduce steps:

1. ./mach run --enable-crash-reporter on Windows
2. Go to https://webvr.info/samples/04-simple-mirroring.html, you will see a VR process is launched.
3. Using MSVC debugger or gdb to attach the parent process.
4. Kill VR process in Process Explorer.
5. The breakpoint will be triggered at CrashReporter::TakeMinidumpForChild() in the parent process.

:gsvelto, I wish you can help answer this. Thanks.

Comment 11

[Gabriele Svelto [:gsvelto]]

Sorry for the delay, today I had an issue with my Windows build (which turned out to be bug 1520928) and I couldn't get a working build until now. Will try to test this ASAP.

Comment 12

[Daosheng Mu[:daoshengmu]]

Try looks good, https://treeherder.mozilla.org/#/jobs?repo=try&revision=6dee07bb497a88b67ae568505a614da8564c1c26.

Comment 13

[Gabriele Svelto [:gsvelto]]

I finally managed to repro the issue you described in comment 9. I made sure that the VR process was launched by visisting that webpage then I killed it. First the good news: the minidump is being generated correctly and it contains everything it should contain. Then the bad news: somehow after the minidump has been fully written out something causes the main process to hang which is probably a side-effect of the pidToMinidump entry being empty. I'll have to investigate further, leaving the NI? for now. BTW the tests are running fine on the other hand.

Comment 14

[Gabriele Svelto [:gsvelto]]

I've checked the entire crash-reporting path and it's working correctly. Not only the minidump is being generated but it's being added correctly to the crash manager and we're even sending a crash ping... That being said the main process locks up when the VR process crashes. I don't know why that's the case but I don't think it's related to the crash reporting code. Is the front-end code aware that the VR process is gone? Other process types inform it one way or another, e.g.:

https://searchfox.org/mozilla-central/rev/dac799c9f4e9f5f05c1071cba94f2522aa31f7eb/dom/ipc/ContentParent.cpp#1689

Which is then handled within the front-end:

https://searchfox.org/mozilla-central/rev/dac799c9f4e9f5f05c1071cba94f2522aa31f7eb/browser/modules/ContentCrashHandlers.jsm#105

BTW I've tested this without VR hardware so YMMV.

Comment 15

[Daosheng Mu[:daoshengmu]]

To run this in a local machine:

1. ./mach run --enable-crash-reporter on Windows
2. Go to https://webvr.info/samples/04-simple-mirroring.html, you will see a VR process is launched.
3. Using MSVC debugger or gdb to attach the parent process.
4. Using crashfirefox-intentionally[1] to kill VR process, the pid can be found in Process Explorer. crashfirefox-intentionally has to be built with MSVC x64 Release setting, and it will generate crashfirefox.exe, type crashfirefox.exe pid in your terminal.
5. The breakpoint will be triggered at CrashReporter::TakeMinidumpForChild() in the parent process.

[1] https://github.com/bsmedberg/crashfirefox-intentionally

Comment 16

[Daosheng Mu[:daoshengmu]]

I just aware if we have no physical headset be connected. After the VR process is killed, the Firefox will be hanged and can't do any control. I will give a part 3 patch to fix it by destroying VR system managers when VR process is shutdown.

Comment 17

[Daosheng Mu[:daoshengmu]]

Attachment: Bug 1518895 - Part 3: Removing VR system managers when VR process is shutdown.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/925142f4e19e
Part 1: Add crash report test for VR process. r=kip,gsvelto
https://hg.mozilla.org/integration/autoland/rev/d3fc75eaf960
Part 2: Crash report support in VR process. r=kip,gsvelto
https://hg.mozilla.org/integration/autoland/rev/624c6d9e882e
Part 3: Removing VR system managers when VR process is shutdown. r=kip

Comment 19

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/925142f4e19e
https://hg.mozilla.org/mozilla-central/rev/d3fc75eaf960
https://hg.mozilla.org/mozilla-central/rev/624c6d9e882e