Closed Bug 1481009 Opened 6 years ago Closed 6 years ago

Support crash reporter in recording/replaying processes

Categories

(Core Graveyard :: Web Replay, enhancement)

Product:
Core Graveyard
Component:
Web Replay
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox63 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox63 --- fixed

People

(Reporter: bhackett1024, Assigned: bhackett1024)

References

Details

Attachments

(9 files)

patch
32.43 KB, patch
Details | Diff | Splinter Review
Part 1 - Enable the crash reporter exception handler in recording, replaying, and middleman processes.
1013 bytes, patch
gsvelto
: review+
Details | Diff | Splinter Review
Part 2 - Avoid changing exception ports when installing the exception handler in replaying processes.
3.71 KB, patch
gsvelto
: review+
Details | Diff | Splinter Review
Part 3 - Report recording/replaying processes crashes as if they happened in the middleman.
6.57 KB, patch
gsvelto
: review+
Details | Diff | Splinter Review
Part 4 - Add redirection for crash reporter code.
1.66 KB, patch
froydnj
: review+
Details | Diff | Splinter Review
Part 5 - Generate a minidump when reporting a fatal record/replay error.
11.78 KB, patch
froydnj
: review+
Details | Diff | Splinter Review
Part 6 - Cleanly shutdown middleman processes after a recording/replaying child crashes.
2.23 KB, patch
froydnj
: review+
Details | Diff | Splinter Review
Part 7 - Treat minaccessible memory regions after thread stacks as untracked.
5.45 KB, patch
froydnj
: review+
Details | Diff | Splinter Review
Part 8 - Remove unused RecordReplayFatalError message.
2.44 KB, patch
froydnj
: review+
Details | Diff | Splinter Review 
The crash reporter is currently disabled in recording/replaying processes.  Fixing this would be really nice, so we would have a way to keep track of user crashes.
Attached patch patch — — Splinter Review 
Rolled up patch to get the crash reporter working in recording, replaying, and middleman processes.  I'll split this up for review shortly.
Assignee: nobody → bhackett1024
Setup the remote exception handler in recording/replaying and middleman processes.  Events are passed through here so that the associated mach ports and other state are live resources in replaying processes (instead of ones that replay data from the recording).
Attachment #8998043 - Flags: review?(gsvelto)
Replaying processes modify the task's exception ports to install the dirty memory handler it uses to keep track of modified heap data.  The crash reporter has its own exception handler that would take precedence over the dirty memory handler, so this patch avoids installing that handler and spawning its associated thread, and adds an API that the replaying process can call later to generate minidumps.
Attachment #8998044 - Flags: review?(gsvelto)
This patch fixes a couple problems when forwarding crashes via the crash generation server/client from recording/replaying processes to the UI process:

1. Children of the middleman process are trying to send their crashes to a crash generation server in the middleman process (which is a content process, and does not have one), instead of the UI process.  This updates the name of the crash notification pipe in the middleman process to use the right pid.

2. Crash minidumps received by the UI process are associated with the pid of the crashing recording/replaying child.  When the middleman shuts down later we'll try to look for a minidump for the middleman process itself, and not find anything.  The UI process doesn't keep track of relationships between middleman pids and their recording/replaying child pids, and rather than add such a thing it seems better to associate minidumps for the recording/replaying children with the middleman itself.
Attachment #8998045 - Flags: review?(gsvelto)
Add a redirection so that some of the crash reporter code that now runs will work.  A new thread was trying to set its name and we ended up crashing due to not redirecting all the associated mach APIs.
Attachment #8998046 - Flags: review?(nfroyd)
Generate minidumps on all record/replay fatal errors.  This includes any crashes in replaying processes which go through the dirty memory handler first; in recording processes these crashes are caught by the crash reporter's own exception handler, which is installed as in normal content processes.
Attachment #8998047 - Flags: review?(nfroyd)
When a recording/replaying child process crashes we don't want the middleman to crash as a result, since it will generate a minidump that masks the one which the child process produced.  This patch fixes a few spots so that the middleman shuts down cleanly after its child process crashes.  The UI process will interpret the middleman's spontaneous shutdown as a crash and will associate it with the minidump produced by the child process.
Attachment #8998049 - Flags: review?(nfroyd)
This patch fixes an unfortunate problem arising from the memory snapshot mechanism in replaying processes.  Replaying processes mark all memory they are tracking as readable, even though some of this memory was originally non-readable when the first checkpoint was reached.  Such memory includes sentinel pages which are placed after each thread's stack, and marking these sentinel pages as readable confuses the crash generation server which runs in the UI process and inspects the crashing process' address space to determine the locations of its thread stacks.  The crash generation server stops being able to distinguish thread stacks which are mapped in adjacent regions of memory, and generates stacks that are over 100 MB instead of a few KB.

This patch fixes this problem by treating the sentinel pages as untracked memory, so that their protection is not modified.
Attachment #8998052 - Flags: review?(nfroyd)
PContent::RecordReplayFatalError is not used anymore and can be removed.  If a fatal error occurs we'll produce a crash report with information about the error, so we can just let the tab crash in the usual way.
Attachment #8998054 - Flags: review?(nfroyd)
Attachment #8998043 - Flags: review?(gsvelto) → review+
Comment on attachment 8998044 [details] [diff] [review]
Part 2 - Avoid changing exception ports when installing the exception handler in replaying processes.

Review of attachment 8998044 [details] [diff] [review]:
-----------------------------------------------------------------

Just a nit, the rest LGTM

::: toolkit/crashreporter/breakpad-client/mac/handler/exception_handler.cc
@@ +645,5 @@
> +						       int exception_subcode,
> +						       mach_port_t thread)
> +{
> +  if (!gProtectedData.handler)
> +    return false;

nit: Missing braces
Attachment #8998044 - Flags: review?(gsvelto) → review+
Comment on attachment 8998045 [details] [diff] [review]
Part 3 - Report recording/replaying processes crashes as if they happened in the middleman.

Review of attachment 8998045 [details] [diff] [review]:
-----------------------------------------------------------------

Looking good overall, just a minor nit

::: toolkit/crashreporter/nsExceptionHandler.cpp
@@ +3335,5 @@
>      true,
>      &dumpPath);
>  
>  #elif defined(XP_MACOSX)
> +  // Middleman processes do not have their own crash generation server.

nit: Could you split this out to a separate function? Something like:

static bool
MaybeForwardIfMiddleman() {
  // Middleman processes ...
}

... and then:

if (MaybeForwardIfMiddleman()) {
  return;
}

Just in the name of making this (already complicated) code more readable :)
Attachment #8998045 - Flags: review?(gsvelto) → review+
Attachment #8998046 - Flags: review?(nfroyd) → review+
Attachment #8998054 - Flags: review?(nfroyd) → review+
Attachment #8998049 - Flags: review?(nfroyd) → review+
Attachment #8998047 - Flags: review?(nfroyd) → review+
Comment on attachment 8998052 [details] [diff] [review]
Part 7 - Treat minaccessible memory regions after thread stacks as untracked.

Review of attachment 8998052 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/recordreplay/MemorySnapshot.cpp
@@ +729,5 @@
>  }
>  
> +// Get information about the mapped region containing *aAddress, or the next
> +// mapped region afterwards if aAddress is not mapped. Returns false if there
> +// are no more mapped regions after aAddress.

This is the weirdest contract for a function, but AFAICT it is the contract for mach_vm_region.

Can this describe what gets returned in aAddress on success?

@@ +783,5 @@
> +    // Find the mapped region containing the thread's stack.
> +    uint8_t* base = thread->StackBase();
> +    size_t size;
> +    if (!QueryRegion(&base, &size)) {
> +      MOZ_CRASH("MarkThreadStacksAsUntracked");

Nit: Maybe "Could not find memory region information about the thread's stack"?
Attachment #8998052 - Flags: review?(nfroyd) → review+
Blocks: 1482275
Pushed by bhackett@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1ec7f635f4f3
Part 1 - Enable the crash reporter exception handler in recording, replaying, and middleman processes, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/58598aa3cf0c
Part 2 - Avoid changing exception ports when installing the exception handler in replaying processes, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/f7f4127aa9d8
Part 3 - Report recording/replaying processes crashes as if they happened in the middleman, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/85eec9cfc3d9
Part 4 - Add redirection for crash reporter code, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/6869ea5ebd32
Part 5 - Generate a minidump when reporting a fatal record/replay error, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e6908123a97
Part 6 - Cleanly shutdown middleman processes after a recording/replaying child crashes, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/4a0c7dd5e1ac
Part 7 - Treat inaccessible memory regions after thread stacks as untracked, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/11ee868c5903
Part 8 - Remove unused RecordReplayFatalError message, r=froydnj.
Backed out for causing devtools crashes @XUL + 0x37bba19. This seems to be failling only on OS X. 

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=pending&filter-resultStatus=running&filter-resultStatus=success&filter-searchStr=OS%20X%2010.10%20opt%20Mochitests%20with%20e10s%20test-macosx64%2Fopt-mochitest-devtools-chrome-e10s-1%20M-e10s(dt1)&fromchange=39ad7ffcd976cb51d908f139c9d19f1e112ee4f5&tochange=935abf820aeb9ef35967b08ae9a27670b2e48a54&selectedJob=193512806

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=193512806&repo=mozilla-inbound&lineNumber=28853

Backout link: https://hg.mozilla.org/mozilla-central/rev/e3cec7443adffef026b98c41bb709a4c48baecb9

03:37:14     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ XUL + 0x37bba19]
03:37:14     INFO - Crash dump filename: /var/folders/nk/lkwxjm4s2zbgych5wk8fk3m000000w/T/tmpzekCIJ.mozrunner/minidumps/AA8FCBDD-3D17-47F1-B930-A8F8DBBB69D0.dmp
03:37:14     INFO - Operating system: Mac OS X
03:37:14     INFO -                   10.10.5 14F27
03:37:14     INFO - CPU: amd64
03:37:14     INFO -      family 6 model 69 stepping 1
03:37:14     INFO -      4 CPUs
03:37:14     INFO - 
03:37:14     INFO - GPU: UNKNOWN
03:37:14     INFO - 
03:37:14     INFO - Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
03:37:14     INFO - Crash address: 0x0
03:37:14     INFO - Process uptime: 613 seconds
03:37:14     INFO - 
03:37:14     INFO - Thread 0 (crashed)
03:37:14     INFO -  0  XUL + 0x37bba19
03:37:14     INFO -     rax = 0x000000011327d424   rdx = 0x0000000000000000
03:37:14     INFO -     rcx = 0x0000000117f9e558   rbx = 0x0000000000000001
03:37:14     INFO -     rsi = 0x0000000000000002   rdi = 0x000000011327d40b
03:37:14     INFO -     rbp = 0x00007fff51983b70   rsp = 0x00007fff51983b70
03:37:14     INFO -      r8 = 0x000000002a519100    r9 = 0x0000000129a2c000
03:37:14     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000001
03:37:14     INFO -     r12 = 0x000000012a0f7c40   r13 = 0x0000000000000046
03:37:14     INFO -     r14 = 0x0000000000000001   r15 = 0x00007fff51983c60
03:37:14     INFO -     rip = 0x0000000111d61a19
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO -  1  XUL + 0x8a2bf8
03:37:14     INFO -     rbp = 0x00007fff51983b80   rsp = 0x00007fff51983b80
03:37:14     INFO -     rip = 0x000000010ee48bf8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  2  XUL + 0xd170e
03:37:14     INFO -     rbp = 0x00007fff51983bb0   rsp = 0x00007fff51983b90
03:37:14     INFO -     rip = 0x000000010e67770e
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  3  XUL + 0x8ce5f1
03:37:14     INFO -     rbp = 0x00007fff51983da0   rsp = 0x00007fff51983bc0
03:37:14     INFO -     rip = 0x000000010ee745f1
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  4  XUL + 0x8cfb16
03:37:14     INFO -     rbp = 0x00007fff51983ed0   rsp = 0x00007fff51983db0
03:37:14     INFO -     rip = 0x000000010ee75b16
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  5  XUL + 0x3950180
03:37:14     INFO -     rbp = 0x00007fff51983f90   rsp = 0x00007fff51983ee0
03:37:14     INFO -     rip = 0x0000000111ef6180
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  6  XUL + 0x39507f9
03:37:14     INFO -     rbp = 0x00007fff51983fb0   rsp = 0x00007fff51983fa0
03:37:14     INFO -     rip = 0x0000000111ef67f9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  7  XUL + 0x3dc0607
03:37:14     INFO -     rbp = 0x00007fff519844b0   rsp = 0x00007fff51983fc0
03:37:14     INFO -     rip = 0x0000000112366607
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  8  XUL + 0x3950180
03:37:14     INFO -     rbp = 0x00007fff51984570   rsp = 0x00007fff519844c0
03:37:14     INFO -     rip = 0x0000000111ef6180
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  9  XUL + 0x394a4e8
03:37:14     INFO -     rbp = 0x00007fff51984a00   rsp = 0x00007fff51984580
03:37:14     INFO -     rip = 0x0000000111ef04e8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 10  XUL + 0x39422c8
03:37:14     INFO -     rbp = 0x00007fff51984af0   rsp = 0x00007fff51984a10
03:37:14     INFO -     rip = 0x0000000111ee82c8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 11  XUL + 0x39503a6
03:37:14     INFO -     rbp = 0x00007fff51984bb0   rsp = 0x00007fff51984b00
03:37:14     INFO -     rip = 0x0000000111ef63a6
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 12  XUL + 0x39507f9
03:37:14     INFO -     rbp = 0x00007fff51984bd0   rsp = 0x00007fff51984bc0
03:37:14     INFO -     rip = 0x0000000111ef67f9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 13  XUL + 0x3cf846a
03:37:14     INFO -     rbp = 0x00007fff51984d10   rsp = 0x00007fff51984be0
03:37:14     INFO -     rip = 0x000000011229e46a
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 14  XUL + 0x3cf1ca8
03:37:14     INFO -     rbp = 0x00007fff51984d60   rsp = 0x00007fff51984d20
03:37:14     INFO -     rip = 0x0000000112297ca8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 15  XUL + 0x3950526
03:37:14     INFO -     rbp = 0x00007fff51984e20   rsp = 0x00007fff51984d70
03:37:14     INFO -     rip = 0x0000000111ef6526
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 16  XUL + 0x39507f9
03:37:14     INFO -     rbp = 0x00007fff51984e40   rsp = 0x00007fff51984e30
03:37:14     INFO -     rip = 0x0000000111ef67f9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 17  XUL + 0x3cfa028
03:37:14     INFO -     rbp = 0x00007fff51984f10   rsp = 0x00007fff51984e50
03:37:14     INFO -     rip = 0x00000001122a0028
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 18  XUL + 0x3ce75f7
03:37:14     INFO -     rbp = 0x00007fff51984f70   rsp = 0x00007fff51984f20
03:37:14     INFO -     rip = 0x000000011228d5f7
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 19  XUL + 0x3cf1ca8
03:37:14     INFO -     rbp = 0x00007fff51984fc0   rsp = 0x00007fff51984f80
03:37:14     INFO -     rip = 0x0000000112297ca8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 20  XUL + 0x3950526
03:37:14     INFO -     rbp = 0x00007fff51985080   rsp = 0x00007fff51984fd0
03:37:14     INFO -     rip = 0x0000000111ef6526
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 21  XUL + 0x394a4e8
03:37:14     INFO -     rbp = 0x00007fff51985510   rsp = 0x00007fff51985090
03:37:14     INFO -     rip = 0x0000000111ef04e8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 22  XUL + 0x39422c8
03:37:14     INFO -     rbp = 0x00007fff51985600   rsp = 0x00007fff51985520
03:37:14     INFO -     rip = 0x0000000111ee82c8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 23  XUL + 0x39503a6
03:37:14     INFO -     rbp = 0x00007fff519856c0   rsp = 0x00007fff51985610
03:37:14     INFO -     rip = 0x0000000111ef63a6
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 24  XUL + 0x39507f9
03:37:14     INFO -     rbp = 0x00007fff519856e0   rsp = 0x00007fff519856d0
03:37:14     INFO -     rip = 0x0000000111ef67f9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 25  XUL + 0x3ca07f6
03:37:14     INFO -     rbp = 0x00007fff519857c0   rsp = 0x00007fff519856f0
03:37:14     INFO -     rip = 0x00000001122467f6
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 26  XUL + 0x17db039
03:37:14     INFO -     rbp = 0x00007fff519858f0   rsp = 0x00007fff519857d0
03:37:14     INFO -     rip = 0x000000010fd81039
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 27  XUL + 0xf5bda4
03:37:14     INFO -     rbp = 0x00007fff51985b30   rsp = 0x00007fff51985900
03:37:14     INFO -     rip = 0x000000010f501da4
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 28  XUL + 0xf5b7ab
03:37:14     INFO -     rbp = 0x00007fff51985de0   rsp = 0x00007fff51985b40
03:37:14     INFO -     rip = 0x000000010f5017ab
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 29  XUL + 0x1008b8b
03:37:14     INFO -     rbp = 0x00007fff51985ed0   rsp = 0x00007fff51985df0
03:37:14     INFO -     rip = 0x000000010f5aeb8b
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 30  XUL + 0x1004431
03:37:14     INFO -     rbp = 0x00007fff51985f00   rsp = 0x00007fff51985ee0
03:37:14     INFO -     rip = 0x000000010f5aa431
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 31  XUL + 0x10046a3
03:37:14     INFO -     rbp = 0x00007fff51985f10   rsp = 0x00007fff51985f10
03:37:14     INFO -     rip = 0x000000010f5aa6a3
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 32  XUL + 0xccefb
03:37:14     INFO -     rbp = 0x00007fff51985fc0   rsp = 0x00007fff51985f20
03:37:14     INFO -     rip = 0x000000010e672efb
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 33  XUL + 0xba7f8
03:37:14     INFO -     rbp = 0x00007fff51985ff0   rsp = 0x00007fff51985fd0
03:37:14     INFO -     rip = 0x000000010e6607f8
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 34  XUL + 0xc8c51
03:37:14     INFO -     rbp = 0x00007fff51986030   rsp = 0x00007fff51986000
03:37:14     INFO -     rip = 0x000000010e66ec51
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 35  XUL + 0xc8a9d
03:37:14     INFO -     rbp = 0x00007fff51986040   rsp = 0x00007fff51986040
03:37:14     INFO -     rip = 0x000000010e66ea9d
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 36  XUL + 0xb086f
03:37:14     INFO -     rbp = 0x00007fff519860d0   rsp = 0x00007fff51986050
03:37:14     INFO -     rip = 0x000000010e65686f
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 37  XUL + 0xc0b9c
03:37:14     INFO -     rbp = 0x00007fff51986610   rsp = 0x00007fff519860e0
03:37:14     INFO -     rip = 0x000000010e666b9c
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 38  XUL + 0xc3577
03:37:14     INFO -     rbp = 0x00007fff51986630   rsp = 0x00007fff51986620
03:37:14     INFO -     rip = 0x000000010e669577
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 39  XUL + 0x5b0709
03:37:14     INFO -     rbp = 0x00007fff51986680   rsp = 0x00007fff51986640
03:37:14     INFO -     rip = 0x000000010eb56709
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 40  XUL + 0x57553b
03:37:14     INFO -     rbp = 0x00007fff519866b0   rsp = 0x00007fff51986690
03:37:14     INFO -     rip = 0x000000010eb1b53b
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 41  XUL + 0x248a8e9
03:37:14     INFO -     rbp = 0x00007fff519866d0   rsp = 0x00007fff519866c0
03:37:14     INFO -     rip = 0x0000000110a308e9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 42  XUL + 0x24f1e0f
03:37:14     INFO -     rbp = 0x00007fff51986700   rsp = 0x00007fff519866e0
03:37:14     INFO -     rip = 0x0000000110a97e0f
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 43  XUL + 0x3816eb5
03:37:14     INFO -     rbp = 0x00007fff51986740   rsp = 0x00007fff51986710
03:37:14     INFO -     rip = 0x0000000111dbceb5
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 44  XUL + 0x57553b
03:37:14     INFO -     rbp = 0x00007fff51986770   rsp = 0x00007fff51986750
03:37:14     INFO -     rip = 0x000000010eb1b53b
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 45  XUL + 0x3816c52
03:37:14     INFO -     rbp = 0x00007fff51986a40   rsp = 0x00007fff51986780
03:37:14     INFO -     rip = 0x0000000111dbcc52
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 46  plugin-container!main [plugin-container.cpp:ffc96a72ae14f9dcd9ba0245cc6f8f9a6dcb6e81 : 50 + 0x13]
03:37:14     INFO -     rbp = 0x00007fff51986a80   rsp = 0x00007fff51986a50
03:37:14     INFO -     rip = 0x000000010e278f39
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 47  libdyld.dylib + 0x35c9
03:37:14     INFO -     rbp = 0x00007fff51986a98   rsp = 0x00007fff51986a90
03:37:14     INFO -     rip = 0x00007fff92af25c9
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 48  libdyld.dylib + 0x35c9
03:37:14     INFO -     rbp = 0x00007fff51986a98   rsp = 0x00007fff51986a98
03:37:14     INFO -     rip = 0x00007fff92af25c9
03:37:14     INFO -     Found by: stack scanning
03:37:14     INFO - 
03:37:14     INFO - Thread 1
03:37:14     INFO -  0  0x11824f0d1
03:37:14     INFO -     rax = 0x0000000002000003   rdx = 0x0000000000000001
03:37:14     INFO -     rcx = 0x000000011803a318   rbx = 0x00000000000000ae
03:37:14     INFO -     rsi = 0x000000011803a32f   rdi = 0x00000000000000ae
03:37:14     INFO -     rbp = 0x000000011803a340   rsp = 0x000000011803a318
03:37:14     INFO -      r8 = 0x0000000000002003    r9 = 0x0000000000000000
03:37:14     INFO -     r10 = 0x000000011824f326   r11 = 0x0000000000000246
03:37:14     INFO -     r12 = 0x0000000000000000   r13 = 0x0000000000002003
03:37:14     INFO -     r14 = 0x000000011803a32f   r15 = 0x0000000000007103
03:37:14     INFO -     rip = 0x000000011824f0d1
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO -  1  XUL + 0x37e8f89
03:37:14     INFO -     rbp = 0x000000011803a350   rsp = 0x000000011803a350
03:37:14     INFO -     rip = 0x0000000111d8ef89
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  2  XUL + 0x37ce4e4
03:37:14     INFO -     rbp = 0x000000011803a3d0   rsp = 0x000000011803a360
03:37:14     INFO -     rip = 0x0000000111d744e4
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  3  libsystem_kernel.dylib + 0x6319
03:37:14     INFO -     rbp = 0x000000011803a430   rsp = 0x000000011803a3e0
03:37:14     INFO -     rip = 0x00007fff9751c319
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  4  libdispatch.dylib + 0x9890
03:37:14     INFO -     rbp = 0x000000011803a480   rsp = 0x000000011803a440
03:37:14     INFO -     rip = 0x00007fff97c6e890
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  5  libdispatch.dylib + 0x5b87
03:37:14     INFO -     rbp = 0x000000011803a4a0   rsp = 0x000000011803a490
03:37:14     INFO -     rip = 0x00007fff97c6ab87
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  6  libdispatch.dylib + 0x5b3a
03:37:14     INFO -     rbp = 0x000000011803a4c0   rsp = 0x000000011803a4b0
03:37:14     INFO -     rip = 0x00007fff97c6ab3a
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  7  libdispatch.dylib + 0x5976
03:37:14     INFO -     rbp = 0x000000011803a4e0   rsp = 0x000000011803a4d0
03:37:14     INFO -     rip = 0x00007fff97c6a976
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  8  libdispatch.dylib + 0x55d3
03:37:14     INFO -     rbp = 0x000000011803a520   rsp = 0x000000011803a4f0
03:37:14     INFO -     rip = 0x00007fff97c6a5d3
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  9  libdispatch.dylib + 0x5154
03:37:14     INFO -     rbp = 0x000000011803a590   rsp = 0x000000011803a530
03:37:14     INFO -     rip = 0x00007fff97c6a154
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 10  libdispatch.dylib + 0x4e0c
03:37:14     INFO -     rbp = 0x000000011803a5b0   rsp = 0x000000011803a5a0
03:37:14     INFO -     rip = 0x00007fff97c69e0c
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 11  libdispatch.dylib + 0x4cd0
03:37:14     INFO -     rbp = 0x000000011803a640   rsp = 0x000000011803a5c0
03:37:14     INFO -     rip = 0x00007fff97c69cd0
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 12  libdispatch.dylib + 0x4a6a
03:37:14     INFO -     rbp = 0x000000011803ae50   rsp = 0x000000011803a650
03:37:14     INFO -     rip = 0x00007fff97c69a6a
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 
03:37:14     INFO - Thread 2
03:37:14     INFO -  0  libsystem_kernel.dylib + 0x1694a
03:37:14     INFO -     rax = 0x0000000002000170   rdx = 0x0000000000000000
03:37:14     INFO -     rcx = 0x00000001180bdf18   rbx = 0x0000000000010000
03:37:14     INFO -     rsi = 0x0000000000000000   rdi = 0x0000000000000004
03:37:14     INFO -     rbp = 0x00000001180bdf50   rsp = 0x00000001180bdf18
03:37:14     INFO -      r8 = 0xfffffffffffff000    r9 = 0x00000000ffffffff
03:37:14     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
03:37:14     INFO -     r12 = 0x00000000800008ff   r13 = 0x0000000000001303
03:37:14     INFO -     r14 = 0x00000001180be000   r15 = 0x0000000000000015
03:37:14     INFO -     rip = 0x00007fff9752c94a
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO -  1  libsystem_pthread.dylib + 0x13dd
03:37:14     INFO -     rbp = 0x00000001180bdf78   rsp = 0x00000001180bdf60
03:37:14     INFO -     rip = 0x00007fff953a13dd
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  2  libdispatch.dylib + 0x12f89
03:37:14     INFO -     rsp = 0x00000001180be030   rip = 0x00007fff97c77f89
03:37:14     INFO -     Found by: stack scanning
03:37:14     INFO - 
03:37:14     INFO - Thread 3
03:37:14     INFO -  0  libsystem_kernel.dylib + 0x1694a
03:37:14     INFO -     rax = 0x0000000002000170   rdx = 0x0000000000000000
03:37:14     INFO -     rcx = 0x0000000118380f18   rbx = 0x0000000000010000
03:37:14     INFO -     rsi = 0x0000000000000000   rdi = 0x0000000000000004
03:37:14     INFO -     rbp = 0x0000000118380f50   rsp = 0x0000000118380f18
03:37:14     INFO -      r8 = 0x0000000000000003    r9 = 0x00000001182336d0
03:37:14     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
03:37:14     INFO -     r12 = 0x00000000800008ff   r13 = 0x0000000000001703
03:37:14     INFO -     r14 = 0x0000000118381000   r15 = 0x0000000000000015
03:37:14     INFO -     rip = 0x00007fff9752c94a
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO -  1  libsystem_pthread.dylib + 0x13dd
03:37:14     INFO -     rbp = 0x0000000118380f78   rsp = 0x0000000118380f60
03:37:14     INFO -     rip = 0x00007fff953a13dd
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  2  libdispatch.dylib + 0x12f89
03:37:14     INFO -     rsp = 0x0000000118381030   rip = 0x00007fff97c77f89
03:37:14     INFO -     Found by: stack scanning
03:37:14     INFO - 
03:37:14     INFO - Thread 4
03:37:14     INFO -  0  libsystem_kernel.dylib + 0x1694a
03:37:14     INFO -     rax = 0x0000000002000170   rdx = 0x0000000000000000
03:37:14     INFO -     rcx = 0x0000000118c80f18   rbx = 0x0000000000010000
03:37:14     INFO -     rsi = 0x0000000000000000   rdi = 0x0000000000000004
03:37:14     INFO -     rbp = 0x0000000118c80f50   rsp = 0x0000000118c80f18
03:37:14     INFO -      r8 = 0x0000000118200000    r9 = 0x00000001182007d8
03:37:14     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
03:37:14     INFO -     r12 = 0x00000000800008ff   r13 = 0x0000000000001407
03:37:14     INFO -     r14 = 0x0000000118c81000   r15 = 0x0000000000000015
03:37:14     INFO -     rip = 0x00007fff9752c94a
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO - 
03:37:14     INFO - Thread 5
03:37:14     INFO -  0  0x11824f0d1
03:37:14     INFO -     rax = 0x0000000002000003   rdx = 0x0000000000000001
03:37:14     INFO -     rcx = 0x0000000118e837a8   rbx = 0x0000000000000009
03:37:14     INFO -     rsi = 0x0000000118e837b4   rdi = 0x0000000000000009
03:37:14     INFO -     rbp = 0x0000000118e837e0   rsp = 0x0000000118e837a8
03:37:14     INFO -      r8 = 0x0000000118e837b0    r9 = 0x00000001180c1238
03:37:14     INFO -     r10 = 0x00007fff9752c6fa   r11 = 0x0000000000000202
03:37:14     INFO -     r12 = 0x0000000117f9e582   r13 = 0x00000001180c0000
03:37:14     INFO -     r14 = 0x0000000118e837b4   r15 = 0x00000001182731b0
03:37:14     INFO -     rip = 0x000000011824f0d1
03:37:14     INFO -     Found by: given as instruction pointer in context
03:37:14     INFO -  1  XUL + 0x37b8285
03:37:14     INFO -     rbp = 0x0000000118e83830   rsp = 0x0000000118e837f0
03:37:14     INFO -     rip = 0x0000000111d5e285
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  2  XUL + 0x37c2fe0
03:37:14     INFO -     rbp = 0x0000000118e83ca0   rsp = 0x0000000118e83840
03:37:14     INFO -     rip = 0x0000000111d68fe0
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  3  XUL + 0x37b6cbc
03:37:14     INFO -     rbp = 0x0000000118e83d80   rsp = 0x0000000118e83cb0
03:37:14     INFO -     rip = 0x0000000111d5ccbc
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  4  XUL + 0x37ce5ae
03:37:14     INFO -     rbp = 0x0000000118e83e00   rsp = 0x0000000118e83d90
03:37:14     INFO -     rip = 0x0000000111d745ae
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  5  XUL + 0x58957d
03:37:14     INFO -     rbp = 0x0000000118e83e20   rsp = 0x0000000118e83e10
03:37:14     INFO -     rip = 0x000000010eb2f57d
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  6  XUL + 0x5bc282
03:37:14     INFO -     rbp = 0x0000000118e83eb0   rsp = 0x0000000118e83e30
03:37:14     INFO -     rip = 0x000000010eb62282
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  7  XUL + 0x37e8823
03:37:14     INFO -     rbp = 0x0000000118e83ef0   rsp = 0x0000000118e83ec0
03:37:14     INFO -     rip = 0x0000000111d8e823
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  8  libsystem_pthread.dylib + 0x405a
03:37:14     INFO -     rbp = 0x0000000118e83f10   rsp = 0x0000000118e83f00
03:37:14     INFO -     rip = 0x00007fff953a405a
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO -  9  libsystem_pthread.dylib + 0x3fd7
03:37:14     INFO -     rbp = 0x0000000118e83f50   rsp = 0x0000000118e83f20
03:37:14     INFO -     rip = 0x00007fff953a3fd7
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 10  libsystem_pthread.dylib + 0x13ed
03:37:14     INFO -     rbp = 0x0000000118e83f78   rsp = 0x0000000118e83f60
03:37:14     INFO -     rip = 0x00007fff953a13ed
03:37:14     INFO -     Found by: previous frame's frame pointer
03:37:14     INFO - 11  XUL + 0x37e87b0
03:37:14     INFO -     rsp = 0x0000000118e84030   rip = 0x0000000111d8e7b0
03:37:14     INFO -     Found by: stack scanning
Status: RESOLVED → REOPENED
status-firefox63: fixed → ---
Flags: needinfo?(bhackett1024)
Resolution: FIXED → ---
Target Milestone: mozilla63 → ---
Pushed by bhackett@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2add7d6141ec
Part 1 - Enable the crash reporter exception handler in recording, replaying, and middleman processes, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b2072d947e6
Part 2 - Avoid changing exception ports when installing the exception handler in replaying processes, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/7927150efaa6
Part 3 - Report recording/replaying processes crashes as if they happened in the middleman, r=gsvelto.
https://hg.mozilla.org/integration/mozilla-inbound/rev/d605332894bf
Part 4 - Add redirection for crash reporter code, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/305a6de8a457
Part 5 - Generate a minidump when reporting a fatal record/replay error, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/41d3f63a86da
Part 6 - Cleanly shutdown middleman processes after a recording/replaying child crashes, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/a3bd72355db1
Part 7 - Treat inaccessible memory regions after thread stacks as untracked, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/cb8d7e42139e
Part 8 - Remove unused RecordReplayFatalError message, r=froydnj.
https://hg.mozilla.org/integration/mozilla-inbound/rev/515c80e08cb9
Disable Web Replay recovery test.
The problem here seems to be something of an own goal.  Shortly before the earlier push I landed the remaining web replay tests, one of which triggers a crash in a replaying process in order to test the middleman's ability to start up a new replaying process and recover it to the correct point.  That crash was dutifully turned into a minidump (whose stacks make no sense, alas) and reported in the treeherder log.  I've disabled the test for now, until a better workaround is found.
Flags: needinfo?(bhackett1024)
Depends on: 1483365
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: