1436505 - Crash in nsDisplayListBuilder::OutOfFlowDisplayData::ComputeVisibleRectForFrame

Status: RESOLVED WORKSFORME

(Core :: Web Painting, defect, P2)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is
report bp-d9c6839a-43da-4706-9a8f-d04530180207.
=============================================================

Top 10 frames of crashing thread:

0 xul.dll nsDisplayListBuilder::OutOfFlowDisplayData::ComputeVisibleRectForFrame layout/painting/nsDisplayList.h:1487
1 xul.dll nsDisplayListBuilder::MarkOutOfFlowFrameForDisplay layout/painting/nsDisplayList.cpp:1178
2 xul.dll nsDisplayListBuilder::MarkFramesForDisplayList layout/painting/nsDisplayList.cpp:1477
3 xul.dll nsIFrame::MarkAbsoluteFramesForDisplayList layout/generic/nsFrame.cpp:3854
4 xul.dll nsIFrame::BuildDisplayListForChild layout/generic/nsFrame.cpp:3739
5 xul.dll nsTableFrame::GenericTraversal layout/tables/nsTableFrame.cpp:1352
6 xul.dll nsTableFrame::DisplayGenericTablePart layout/tables/nsTableFrame.cpp:1611
7 xul.dll nsTableRowFrame::BuildDisplayList layout/tables/nsTableColFrame.cpp:129
8 xul.dll nsIFrame::BuildDisplayListForChild layout/generic/nsFrame.cpp:3526
9 xul.dll DisplayRows layout/tables/nsTableRowGroupFrame.cpp:234

=============================================================

crash reports with this signature in the content process on windows are starting to show up in low volume during 59.0b, looking perhaps related to bug 1344971.

Comment 1

[Daniel Veditz [:dveditz]]

Assigning unowned critical/high security bugs to triage owner. Please find an appropriate assignee for this bug.

Comment 2

[Matt Woodrow (:mattwoodrow)]

Looking at the minidump, this appears to be a codegen issue?

The crash is on [ebx+0x1C], but nothing sets ebx in that function, or in the callers. In MarkFramesForDisplayList, ebx is used to store the nsDisplayListBuilder, and I can't see any code that writes to it between there and the crash.

This might have disappeared in b8, or it's just too rare to show up yet.

Comment 3

[Frederik Braun [:freddy]]

Matt, this is showing up still, would it make sense to look at more minidumps?

Comment 4

[Gian-Carlo Pascutto [:gcp]]

>Looking at the minidump, this appears to be a codegen issue?

It would have to be a codegen issue that affects both 32-bit and 64-bit compiler backends, which makes this a bit less likely? And it's happening on both macOS and Windows.

So this is likely a real bug.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

So in crash https://crash-stats.mozilla.com/report/index/7d3d2daf-2633-4379-abcc-f2a380180508
(which faults on an e5e5 address), it's getting e5e5 back from aFrame->GetParent(). It crashes trying to do parent->GetParent().  This implies strongly *aFrame has been freed.

This isn't the first bug I've seen (in DisplayList bugs?) where something has freed the frame or it's parent....  That may be the underlying root cause of many bugs/UAFs (and not just here, but also CC/GC, etc).

bz, matt, any thoughts or tripwire/assertion suggestions?  God, I wish we would ship ASAN builds to the field...

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

BTW, in that instance I don't see any likelihood it's a codegen issue

Comment 7

[Boris Zbarsky [:bzbarsky]]

Is this about retained display lists holding refs to dead frames, or frames dying while a "non-retained" display list use is in process?

For the former, if we're willing to do somewhat slow asserts, we could do something where whenever a frame is destroyed we check whether a retained display list is pointing to it (e.g. we could have a map from frames to number of uses to avoid having to walk the whole list).

Comment 8

[Matt Woodrow (:mattwoodrow)]

It looks like the latter.

We're doing a display list build (so not accessing any of the retained structures), we grab GetAbsoluteContainingBlock()->GetChildList() for a frame (that returns true for IsAbsoluteContainer), iterate that list, and one of the entries appears to be dead.

That sounds like a layout issue, where we've deleted a frame but haven't removed it from the containing block. I seem to remember seeing a similar bug recently, but I can't find it again now, will keep looking.

Comment 9

[Matt Woodrow (:mattwoodrow)]

Bug 1452277!

Gerald, do you think this could be similar?

Comment 10

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

In bug 1452277, during reflow an out-of-flow frame is destroyed (because its overflow containers list gets destroyed), and its placeholder has its mOutOfFlowFrame link correctly reset; then during the same reflow, further accesses through the placeholder crash because of its now-null pointer.

This seems different here, with an absolute frame container (not a placeholder) surviving a full reflow, then dying during painting, and not always with a null/near-null address.
Of course it could just be a "lucky" coincidence and some frame was destroyed late in reflow and not accessed again until painting?

So based on my still-limited knowledge, I doubt it's the same issue -- But I could very well be wrong!
Being able to reproduce this would be great, please ping me again if you find good STRs.


In the meantime, someone more experienced should also have a look.
David is traveling... Mats, please?

Comment 11

[Matt Woodrow (:mattwoodrow)]

(In reply to Gerald Squelart [:gerald] from comment #10)

> This seems different here, with an absolute frame container (not a
> placeholder) surviving a full reflow, then dying during painting, and not
> always with a null/near-null address.
> Of course it could just be a "lucky" coincidence and some frame was
> destroyed late in reflow and not accessed again until painting?

We shouldn't ever be deleting frames during painting (or making any modifications at all), so I suspect that it was reflow (or just normal frame construction) that left us in an invalid state for painting to trip over.

I thought it might be possible that we're clearing the mOutOfFlowFrame link from the placeholder, deleting the OOF, but forgetting to remove the (now dangling) pointer from the containing block's absolutely positioned frames list.

It sounds like that's not the case for your bug, but this might be somewhat similar (removing OOFs in complex situations leads to inconsistent state).

Comment 12

[Matt Woodrow (:mattwoodrow)]

I had a look at the minidump, but none of the frame contents are in the dump, so I can't determine anything about the presence of overflow containers (from the state bits).

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

There are no common URLs reported.  Some of these are fairly unlikely like about:sessionrestore, so the URL identified may not be useful.

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

At some perf cost (and pain making the optimizer not optimize it away), you could copy a few bits (or the entire) frame being operated on onto the stack, such that it would be visible in a dump.  (As a diagnostic patch).  This would be something of a last resort if we can't make any progress again.

Comment 15

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Browsing around the code, one thing I've noticed is quite a few NS_ASSERTIONs that would catch some of these unexpected situations (dangling placeholder, non-empty frame tree being destroyed, etc.), so it's not crashing there but later when one of these bad frame is used.
Would it be worth considering turning some/all of these into MOZ_ASSERTs, to get earlier crashes closer to the root cause?

Comment 16

[Matt Woodrow (:mattwoodrow)]

(In reply to Gerald Squelart [:gerald] from comment #15)
> Browsing around the code, one thing I've noticed is quite a few
> NS_ASSERTIONs that would catch some of these unexpected situations (dangling
> placeholder, non-empty frame tree being destroyed, etc.), so it's not
> crashing there but later when one of these bad frame is used.
> Would it be worth considering turning some/all of these into MOZ_ASSERTs, to
> get earlier crashes closer to the root cause?

Maybe MOZ_DIAGNOSTIC_ASSERT even, so that they're present in optimized Nightly builds?

Especially for ones that are low overhead, catching them in our Nightly user base seems useful.

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #16)
> (In reply to Gerald Squelart [:gerald] from comment #15)
> > Browsing around the code, one thing I've noticed is quite a few
> > NS_ASSERTIONs that would catch some of these unexpected situations (dangling
> > placeholder, non-empty frame tree being destroyed, etc.), so it's not
> > crashing there but later when one of these bad frame is used.
> > Would it be worth considering turning some/all of these into MOZ_ASSERTs, to
> > get earlier crashes closer to the root cause?
> 
> Maybe MOZ_DIAGNOSTIC_ASSERT even, so that they're present in optimized
> Nightly builds?

Definitely.  MOZ_ASSERT would be debug-only, and (unfortunately) the only real debug testing that's done is automation, and if automation was hitting NS_ASSERTION it would be flagged in treeherder, right?  (Certainly used to be the case)

> Especially for ones that are low overhead, catching them in our Nightly user
> base seems useful.

Right.  The question is where is it low-enough overhead?  Lots of this I presume is hot-path.  And we could make them nightly-only, or nightly and early-beta.

Comment 18

[Mats Palmgren (inactive)]

(In reply to Randell Jesup [:jesup] from comment #5)
> (which faults on an e5e5 address), it's getting e5e5 back from
> aFrame->GetParent(). It crashes trying to do parent->GetParent().  This
> implies strongly *aFrame has been freed.

Right, it indicates that the PresShell the frame was allocated in has
been freed.  Frames themselves are never really freed while the shell
is alive - they are allocated from an arena, poisoned while not in
use and then recycled as needed.  (I don't think the poison value
can ever be e5e5).

(In reply to Matt Woodrow (:mattwoodrow) from comment #11)
> ... so I suspect that it was reflow (or just normal frame
> construction) that left us in an invalid state for painting to trip over.

Reflow / frame construction can't destroy the shell, AFAIK.

So, the shell in the PresShell::Paint call is either valid and just
happens to have a wild pointer (and by that I mean something that doesn't
point to a frame at all, because if it did we would see the poison value)
in the frame tree, or a frame from a different shell snuck in somehow and
that shell then got freed.  Either seems rather unlikely to me.
Or, the shell itself is already dead when Paint was called on it,
which of course should never occur.  That also seems unlikely given
that we get quite far into the frame tree before crashing.

crash-data indicates this signature first showed up in v59, is that b/c
nsDisplayListBuilder::OutOfFlowDisplayData::ComputeVisibleRectForFrame
is new?  or perhaps we wiped our crash data before then?  If not, then
it seems this is a regression and the first crash could provide some
lead as to what caused it?

Comment 19

[Matt Woodrow (:mattwoodrow)]

Thanks Mats, this raises some really good points!


(In reply to Mats Palmgren (:mats) from comment #18)
> Right, it indicates that the PresShell the frame was allocated in has
> been freed.  Frames themselves are never really freed while the shell
> is alive - they are allocated from an arena, poisoned while not in
> use and then recycled as needed.  (I don't think the poison value
> can ever be e5e5).

This is really interesting, for some reason I was treating e5e5 as the 'everything' poison value, but it's really the mozjemalloc poison value. You're right that we shouldn't ever be using e5e5 for poisoning the arena, so it really does sound like aFrame points to something outside of the arena (or to a free'd arena).

> 
> (In reply to Matt Woodrow (:mattwoodrow) from comment #11)
> > ... so I suspect that it was reflow (or just normal frame
> > construction) that left us in an invalid state for painting to trip over.
> 
> Reflow / frame construction can't destroy the shell, AFAIK.

Right, if it's the whole shell going away (and not just a single frame), then I agree with this.

> 
> So, the shell in the PresShell::Paint call is either valid and just
> happens to have a wild pointer (and by that I mean something that doesn't
> point to a frame at all, because if it did we would see the poison value)
> in the frame tree, or a frame from a different shell snuck in somehow and
> that shell then got freed.  Either seems rather unlikely to me.
> Or, the shell itself is already dead when Paint was called on it,
> which of course should never occur.  That also seems unlikely given
> that we get quite far into the frame tree before crashing.

Agreed that it seems really unlikely that the shell itself is gone, the painting callstack is really deep here, and we've got there by walking down the frame tree. I don't see how we could have done that without the majority of the frame tree still being valid.

One maybe interesting thing is that the callstack (of the report from comment 0) is that the crash is coming from MarkFramesForDisplayList coming from table frames.

This seems very similar to bug 1406727, where I frequently see the crashes happening with table code on the stack (and also flexbox?). I also just did some investigation that lead me towards being suspicious about MarkFramesForDisplayList.

I'll post the results of that in bug 1406727, would really appreciate if you could take a look and see if you have any extra conclusions.

If we did have a pointer to frame from another shell, which is now deleted, then we might see a poisoning crash like this one. We might also see corruption when we write something to that frame (MarkFramesForDisplayList sets a frame state bit!), which would then affect other code that recently allocated memory (bug 1406727 looks like someone is corrupting our display list building arena allocations).

> 
> crash-data indicates this signature first showed up in v59, is that b/c
> nsDisplayListBuilder::OutOfFlowDisplayData::ComputeVisibleRectForFrame
> is new?  or perhaps we wiped our crash data before then?  If not, then
> it seems this is a regression and the first crash could provide some
> lead as to what caused it?

This function name is new (refactored as part of the retained-dl landing), but the code itself is old.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

> If we did have a pointer to frame from another shell, which is now deleted,
> then we might see a poisoning crash like this one. We might also see
> corruption when we write something to that frame (MarkFramesForDisplayList
> sets a frame state bit!), which would then affect other code that recently
> allocated memory (bug 1406727 looks like someone is corrupting our display
> list building arena allocations).

Is there any sort of release assertion we could put in (say nightly only, or nightly-and-early-beta) that would catch such a case?  (usage of debug is basically just automation unfortunately, due to excessive overhead especially from JS heapchecks).  Obviously ASAN builds in the field (once we ship them) might catch this, but will be used by far fewer people).

Comment 21

[Mats Palmgren (inactive)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #19)

We could sprinkle more Check() calls if you think it helps.
Including the overwritten canary value in the assertion might help too.

Fwiw, I looked through the URLs for this signature and bug 1406727,
but I didn't see anything out of the ordinary.

I guess we could also audit the lifetime of the involved objects,
i.e. DisplayListBuilder / DisplayList / DisplayListClipState /
PresShell / frames etc, and see if we can add more assertions
around those.

DisplayListBuilder has mPresShellStates so I'm guessing it contains
pointers into multiple shells, which seems a bit fragile.
Can we add an assertion at the end of PresShell::Destroy that loops
all live DisplayListBuilders and check that they contain zero
references to this shell or any frames in it?

DisplayListClipState contains pointers to objects on the stack IIRC.
Might be worth auditing that once more and consider adding canaries /
assertions around that, or just rewrite it to not do that?

Comment 22

[Mats Palmgren (inactive)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #19)
> This function name is new (refactored as part of the retained-dl landing),
> but the code itself is old.

OK, can you add the old name to the Signatures field too please?

Comment 23

[Matt Woodrow (:mattwoodrow)]

Attachment: Assert frame ownership

How do you feel about trying something like this?

Comment 24

[Mats Palmgren (inactive)]

Comment on attachment 8979422 [details] [diff] [review]
Assert frame ownership

I'm a little concerned about the perf impact of the mAllocatedPointers.Contains calls, but it might be OK to have it in Nightly for a while.  I'd be inclined to revert that change after while though if it doesn't give any results, so please add a code comment there that saying it's temporary?

The other MOZ_DIAGNOSTIC_ASSERT seems OK from a perf perspective so we can probably leave them in permanently (for Nightly).

The MOZ_CRASH_UNSAFE_PRINTF looks fine to me but you should probably get a data-review on that just in case (I don't know exactly how strict that policy is, so I assume it's "for everything").

Comment 25

[Mats Palmgren (inactive)]

I'm a bit skeptical these assertions will find anything though,
but... famous last words... :-)

I think integrity checks on the DisplayList/Item/Builder/Clip data would be
more valuable, but that might be harder to do with acceptable performance.
But MOZ_ASSERTs should be OK, even if they cost a bit of perf.

Comment 26

[Matt Woodrow (:mattwoodrow)]

Comment on attachment 8979422 [details] [diff] [review]
Assert frame ownership

[Security approval request comment]
How easily could an exploit be constructed based on the patch?

Not easily. I'm clutching at straws a bit here with a diagnostic patch, we really don't even know it this is the issue.

Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?

Nope

Which older supported branches are affected by this flaw?

All! Very long standing issue.

Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be?

Planning on landing on mozilla-central only as it's just diagnostic.

How likely is this patch to cause regressions; how much testing does it need?

Unlikely, just enables some debugging code and adds some new assertions.

Comment 27

[Matt Woodrow (:mattwoodrow)]

Comment on attachment 8979422 [details] [diff] [review]
Assert frame ownership

Also requesting r?francois for data-review on the new MOZ_CRASH_UNSAFE_PRINTF. This affects all users of the CorruptionCanary, and prints out the new value that got corrupted. It's possible that this value shouldn't be public, but I don't think there's any easy way to extract useful information when you only get a single pointer-sized value into a crash report.

Comment 28

[Matt Woodrow (:mattwoodrow)]

(In reply to Mats Palmgren (:mats) from comment #25)
> I'm a bit skeptical these assertions will find anything though,
> but... famous last words... :-)

I do agree, but this bug has me grasping at straws. It'd be really weird for something within the display list arena to clobber the header of an arena segment, unless we're just invalid casting and doing a buffer overrun (something I've audited for multiple times). We should never re-use arena memory for something of a different size, so UAF within the arena seems unlikely.

Long shot for sure, but why not!

> 
> I think integrity checks on the DisplayList/Item/Builder/Clip data would be
> more valuable, but that might be harder to do with acceptable performance.
> But MOZ_ASSERTs should be OK, even if they cost a bit of perf.

Hopefully I can learn something more from this round of diagnostics, and then take some inspired guesses as for what to instrument next.

Comment 29

[François Marier [:francois]]

Comment on attachment 8979422 [details] [diff] [review]
Assert frame ownership

(In reply to Matt Woodrow (:mattwoodrow) from comment #27)
> Also requesting r?francois for data-review on the new
> MOZ_CRASH_UNSAFE_PRINTF. This affects all users of the CorruptionCanary, and
> prints out the new value that got corrupted. It's possible that this value
> shouldn't be public, but I don't think there's any easy way to extract
> useful information when you only get a single pointer-sized value into a
> crash report.

Could you please fill out the data review form (https://github.com/mozilla/data-review/blob/master/request.md) and attach it as a text file here?

The critical piece here is #5: describing the data that will be collected in these asserts and whether or not it would have browsing history or PII in it (Categories 3 and 4).

It's both to help me review but also to have a record of why we're adding these asserts.

Thanks!

Comment 30

[Al Billings [:abillings - ex-MoCo]]

sec-approval+ to land the diagnostic patch on trunk.

Comment 31

[Matt Woodrow (:mattwoodrow)]

Reading the data review form made me realise that it's hard to reason about all possible users of CorruptionCanary, and so equally hard for anyone to approve it :)

Landing the patch without that chunk for now. I'll see if those show anything, and if not then I'll try a new version that only records the data for the display list use case.

https://hg.mozilla.org/integration/mozilla-inbound/rev/a0414a6c423bd33ea684d97458eb7b85eb664afe

Comment 33

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/a0414a6c423b

Comment 34

[Mats Palmgren (inactive)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #31)
> Reading the data review form made me realise that it's hard to reason about
> all possible users of CorruptionCanary, and so equally hard for anyone to
> approve it :)

We already collect "Crash Address" which I think is similar from
a data leakage POV (it could contain sensitive data if the stack
was overwritten).  Collecting the canary value (also a pointer-sized
value) doesn't add to what we already collect in any significant
way since it would only be collected for a very small set of crashes
(where the added assertion fails) and for those crashes we remove
the possibility of leaking any data through "Crash Address".

The added risk from adding this assertion temporarily seems
negligible to me.

Comment 35

[Ryan VanderMeulen [:RyanVM]]

Crash hitting one of the new diagnostic asserts:
https://crash-stats.mozilla.com/report/index/681d3a90-5676-401c-b408-bce010180530

Comment 36

[Mats Palmgren (inactive)]

Matt and I discussed that one on IRC.  He checked the minidump and found
that 'aPtr' was 0x1f7205fe4b0, which seems like a plausible valid pointer.
I guess it could be an actual double-free, or the pointer could belong
to a different shell, or the stack or RAM was corrupted somehow.
There was no URL or contact info or user comments in that report.

Comment 37

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 61 and 62, sec high and related to a sec-critical crash that we're also tracking (bug 1406727)

Comment 38

[Ryan VanderMeulen [:RyanVM]]

While I'm still very-much interested in fixing this in 61 if at all possible, I don't think tracking it is going to help at this point. Please don't hesitate to nominate a patch for uplift should the root cause be found and fixed in time.

Comment 39

[Mats Palmgren (inactive)]

One more, from a different user: bp-e33de148-9c9a-4ccb-a9e1-c11850180605

Comment 40

[Mats Palmgren (inactive)]

The URL for the last one appears to be a link to a post in a public forum:
https://boards.fireden.net/vg/thread/216756920/#216774815

It works for me when I load it a few times on Linux...
I wonder if we could set up an automated stress test that browses
these forums like a user would?

Comment 41

[Bob Clary [:bc] (inactive)]

mats: This wouldn't be something run in CI, but something you could run locally on demand?

If so, I would think we could use Marionette to "spider" a url and follow the links in the way the old Spider extension used to do. Is that what you are interested in? Is this something you would like to be available from mach?

Comment 42

[Mats Palmgren (inactive)]

Oh, I thought you already had something to stress test specific URLs.
I guess what would be useful for a crash like this is to spider
any links, but only within the top domain.  Sprinkled with some
scrolling here and there.  Perhaps also open some of the links
in a new tab and then close those some time later.

Meanwhile, I'd appreciate if QA could test that site manually on
platforms other than Linux.

Comment 43

[Mats Palmgren (inactive)]

Given that both crashes came via nsLineBox::DeleteLineList,
I'm beefing up the assertions there in bug 1467526 to hopefully
get some more info.

Comment 44

[Mats Palmgren (inactive)]

There are new MOZ_RELEASE_ASSERT(mAllocatedPointers.Contains(aPtr)) crashes,
most of them seems to come via nsIFrame::RemoveDisplayItemDataForDeletion(), e.g.:
bp-b1043264-50c8-4a3b-a7ea-8ec7b0180621
bp-af159f98-cafc-4224-8340-41d2a0180621
bp-5492043f-d87b-4293-b798-78e470180621
bp-4e303fb9-49a0-4dc1-a258-7ddf50180622
etc.

One is from a nsLineBox::DeleteLineList call though:
bp-393a3d60-c113-4883-b6b6-5c6450180622

Comment 45

[Mats Palmgren (inactive)]

Two different users crashed on the same site, which seems like more than
a coincidence given the low number of crashes.  The site looks like a game
related public wiki, so the URLs are likely not sensitive:
https://de.wiki.elvenar.com/index.php?title=Milit%C3%A4r
https://de.wiki.elvenar.com/index.php?title=Geb%C3%A4ude

Comment 46

[Mats Palmgren (inactive)]

Note that we also got a couple of crashes in nsIPresShell::RecordAlloc
(in bug 1472533):  MOZ_RELEASE_ASSERT(!mAllocatedPointers.Contains(aPtr))
which is pretty ridiculous since we only call RecordAlloc with pointers
we got from a fresh allocation from the arena.  Is the memory used by
these data structures arbitrarily clobbered somehow?

Comment 47

[Liz Henry (:lizzard) (relman/hg->git project)]

Andrei, can your team do some testing as suggested in comment 42?
I'm not sure if you still follow the qawanted tag.

Comment 48

[Liz Henry (:lizzard) (relman/hg->git project)]

I'll keep tracking this, while I'm untracking other potentially related issues.
If we figure this out, we should follow up on bug 1198710, bug 1406727 as well.

Comment 49

[Adrian Florinescu [:aflorinescu]]

Ni'ed Roxana for additional investigation as requested in comment 47.

Comment 50

[roxleitan]

We didn't manage to reproduce the crash on Windows 10 x64, Windows 7 x64, Windows 8.1 and Mac OS X 10.12 with Nightly 62.0a1(2018-06-15), latest Nightly 63.0a1(2018-07-09), Firefox Beta 61.0b12 or Firefox Beta 62.0b6 using the links from comment 40 and comment 45 and following the instructions from comment 42.

Please let us know if any further investigation is needed.

Comment 51

[Liz Henry (:lizzard) (relman/hg->git project)]

We don't have any good leads here, and the layout and sec teams are aware of this and related issues, so I'm going to untrack this for now.

Comment 52

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:mattwoodrow, maybe it's time to close this bug?

Comment 53

[David Lawrence [:dkl]]

Removing employee no longer with company from CC list of private bugs.

Comment 54

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months and this bug has priority 'P2'/severity 'critical'.
:miko, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 55

[Miko Mynttinen]

Last crash was in 91.5.1esr.

Comment 56

[BugBot [:suhaib / :marco/ :calixte]]

Since the bug is closed, the stalled keyword is now meaningless.
For more information, please visit auto_nag documentation.