1276558 - Crash in `anonymous namespace''::TypeAnalyzer::insertConversions

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine: JIT, defect, P3)

Description

[Nicholas Nethercote [inactive]]

This bug was filed from the Socorro interface and is 
report bp-e511ea20-bdd3-428e-8df0-1b6d42160529.
=============================================================

31 crashes with this signature in the past 7 days. Crash address is always 0x0. I looked at a minidump and the crashing instruction was this:

> call qword ptr [r9]

and r9 was 0x0. I'm not sure what this means -- is |iter| null? is adjustInputs somehow null (bad vtable)?

Jan, any ideas?

Comment 1

[Jan de Mooij [:jandem]]

(In reply to Nicholas Nethercote [:njn] from comment #0)
> Crash address is always
> 0x0. I looked at a minidump and the crashing instruction was this:
> 
> > call qword ptr [r9]
> 
> and r9 was 0x0. I'm not sure what this means -- is |iter| null? is
> adjustInputs somehow null (bad vtable)?
> 
> Jan, any ideas?

I looked at bp-9565cba8-aeed-4392-9e78-4c9d82160525.

We're crashing in TypeAnalyzer::adjustInputs. It's inlined so the debugger gets confused. The (virtual) policy->adjustInputs call here crashes:

    if (policy && !policy->adjustInputs(alloc(), ins))
        return false;

The vtable pointer is in rax, that's an address inside libxul. If I look at the memory view, I see many pages filled with zeroes around this address. Not sure what's going on there - it could be a bogus vtable but the address is not entirely bogus..

The list of crashes here shows that the uptime is extremely low for these crashes on Nightly (1, 1, 1, 2, 3, 4 seconds):

https://crash-stats.mozilla.com/signature/?date=%3C2016-05-31T12%3A16%3A34.360318&version=49.0a1&signature=%60anonymous%20namespace%27%27%3A%3ATypeAnalyzer%3A%3AinsertConversions&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=uptime&page=1#reports

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly(version 50):55 crashes from 2016-06-06.
 - aurora (version 49):60 crashes from 2016-06-07.
 - beta   (version 48):62 crashes from 2016-06-06.
 - release(version 47):41 crashes from 2016-05-31.
 - esr    (version 45):5 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly      18       4       1       6       3      11       4
 - aurora       36       5       1       1       3       8       3
 - beta          9       5      10       7       8      10      12
 - release       4       9       4       3       4       4       9
 - esr           0       0       3       1       0       0       0

Affected platform: Windows

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly (version 51): 4 crashes from 2016-08-01.
 - aurora  (version 50): 4 crashes from 2016-08-01.
 - beta    (version 49): 25 crashes from 2016-08-02.
 - release (version 48): 21 crashes from 2016-07-25.
 - esr     (version 45): 8 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       2       1       0
 - aurora        1       1       0
 - beta         10      13       1
 - release       6       5       5
 - esr           0       0       1

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #298
 - aurora  #399
 - beta    #3016     #1919
 - release #2422
 - esr     #2187

Comment 5

[[:philipp]]

this signature seems to spike in early data for the 49.0b10 release, where it's also a startup crash.

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Not great but also not a huge crash volume. I don't think this should be a release blocker for 49.

Comment 7

[[:philipp]]

[Tracking Requested - why for this release]:
this is currently the #3 startup crasher for 49.0.1 (0.36% of all browser crashes, 2% of startup crashes) - could we look into it more trying to get it fixed in upcoming releases?

Comment 8

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Jon, Terrence, given the severity (see comment 7), would you be able to help investigate?

Comment 9

[Terrence Cole [:terrence]]

Do we have reason at all to expect that this is GC related? Unless we have STR, we're not going to be able to tell you anything more than what Jan already discovered in comment 1.

Comment 10

[Andrew McCreight [:mccr8]]

This crash is mostly non-existent, up through 49.0b10, judging by comment 5 and the release management bot posts before that. What landed in that version of beta? Maybe something there is to blame.

Comment 11

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

This link shows all commits included in 49.0b10 https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_49_0b9_RELEASE&tochange=FIREFOX_49_0b10_RELEASE

Comment 12

[Marco Castelluccio [:marco]]

(100.0% in signature vs 05.22% overall) cpu_arch = amd64
(100.0% in signature vs 10.86% overall) address = 0x0
(100.0% in signature vs 34.81% overall) reason = EXCEPTION_ACCESS_VIOLATION_READ
(65.85% in signature vs 25.56% overall) platform_pretty_version = Windows 10
(100.0% in signature vs 67.04% overall) dom_ipc_enabled = null
(43.53% in signature vs 12.88% overall) platform_version = 10.0.14393
(19.87% in signature vs 00.54% overall) cpu_info = family 6 model 58 stepping 9 | 4

"dom_ipc_enabled = null" could be spurious, since the crash is happening
really early at startup.

Comment 13

[Jon Coppeard (:jonco)]

Cancelling my needinfo since this does not appear to be GC related.

Comment 14

[Till Schneidereit [:till]]

I can't imagine anything that landed between b9 and b10 made a difference here. The fact that it's AMD-only is interesting.

Comment 15

[Marco Castelluccio [:marco]]

(In reply to Till Schneidereit [:till] from comment #14)
> I can't imagine anything that landed between b9 and b10 made a difference
> here. The fact that it's AMD-only is interesting.

cpu_arch=amd64 actually means the target architecture of the Firefox
build, so this is Firefox 64-bit only.

Comment 16

[Till Schneidereit [:till]]

(In reply to Marco Castelluccio [:marco] from comment #15)
> cpu_arch=amd64 actually means the target architecture of the Firefox
> build, so this is Firefox 64-bit only.

Ugh, of course. The fact that it's 64bit-only is interesting, too, though :)

Comment 17

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I only see 1 instance of this crash on Beta50 in the past 7 days, don't feel the need to track it and updating the status to wontfix for 50.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly (version 52): 9 crashes from 2016-09-19.
 - aurora  (version 51): 3 crashes from 2016-09-19.
 - beta    (version 50): 6 crashes from 2016-09-20.
 - release (version 49): 4208 crashes from 2016-09-05.
 - esr     (version 45): 10 crashes from 2016-07-25.

Crash volume on the last weeks (Week N is from 10-17 to 10-23):
            W. N-1  W. N-2  W. N-3  W. N-4
 - nightly       4       4       0       0
 - aurora        0       0       2       0
 - beta          2       0       4       0
 - release    1364    1199    1077     161
 - esr           3       1       0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #260      #203
 - aurora  #1321
 - beta    #5238
 - release #33       #2346
 - esr     #2575

Comment 19

[Chris Peterson [:cpeterson]]

This is the top crash for 64-bit Release 49.

Comment 20

[Chris Peterson [:cpeterson]]

@ Liz: were there any changes in 49.0.1 besides the DLL shim for Websense bug 1304783? Was that issue subsequently fixed in 49.0.2?

https://www.mozilla.org/en-US/firefox/49.0.1/releasenotes/

This signature spiked in Release 49.0.1 but only for 64-bit Firefox. Crash reports per 49.0.x dot-release:

Firefox  32-bit  64-bit
=======  ======  ======
 49.0.0       0       1
 49.0.1      12    4963 !!!
 49.0.2       9      56

Comment 21

[Liz Henry (:lizzard) (relman/hg->git project)]

Chris: we only shipped 49.0 to a small segment of the user population (the first day of the release); then we throttled updates and put out 49.0.1. Yes, as far as I can tell, the websense DLL shim was the only change between 49.0 and 49.0.1.  I wouldn't expect to see this crash show up significantly in 49.0. 

And here's the changes for 49.0.2: https://www.mozilla.org/en-US/firefox/49.0.2/releasenotes/ 
Do you think something we added to 49.0.2 fixed this? I still see some crashes for 49.0.2 (and in 50/51)

Comment 22

[Chris Peterson [:cpeterson]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> Chris: we only shipped 49.0 to a small segment of the user population (the
> first day of the release); then we throttled updates and put out 49.0.1.
> Yes, as far as I can tell, the websense DLL shim was the only change between
> 49.0 and 49.0.1.  I wouldn't expect to see this crash show up significantly
> in 49.0. 

I see. So this is probably not a regression in 49.0.1. Comment 5 says this crash first spiked in Beta 49.0b10.

> And here's the changes for 49.0.2:
> https://www.mozilla.org/en-US/firefox/49.0.2/releasenotes/ 
> Do you think something we added to 49.0.2 fixed this? I still see some
> crashes for 49.0.2 (and in 50/51)

None of the changes in 49.0.2 look like obvious fixes. Perhaps this crash is caused by some random 64-bit-only code-generation bug at compile time? I don't really know.

Comment 23

[Gerry Chang [:gchang]]

Track 51- as the volume of crash in 51 is very low.

Comment 24

[[:philipp]]

this is spiking up again in win64 builds of firefox 52.0b6

Comment 25

[Chris Peterson [:cpeterson]]

(In reply to [:philipp] from comment #24)
> this is spiking up again in win64 builds of firefox 52.0b6

Does this look like a regression in 52.0b6 or just coincidental timing? Are these crashes also spiking in 52.0b5 or 51.0 during this same time period?

Comment 26

[[:philipp]]

the spike is just in beta6 - so this bug may well be some random build-specific error that pops up every now & then like you suggested in comment #22

Comment 27

[Chris Peterson [:cpeterson]]

Here are the changes between 52.0b5 and 52.0b6. None of them stand out as likely suspects for causing 52.0b6's spike in JIT crashes.

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_52_0b5_RELEASE&tochange=FIREFOX_52_0b6_RELEASE

Comment 28

[Hannes Verschore [:h4writer]]

(In reply to Chris Peterson [:cpeterson] from comment #27)
> Here are the changes between 52.0b5 and 52.0b6. None of them stand out as
> likely suspects for causing 52.0b6's spike in JIT crashes.
> 
> https://hg.mozilla.org/releases/mozilla-beta/
> pushloghtml?fromchange=FIREFOX_52_0b5_RELEASE&tochange=FIREFOX_52_0b6_RELEASE

I also don't see anything particular for the jits.

Comment 29

[Chris Peterson [:cpeterson]]

Jan, based on your initial investigation in comment 1, is there any diagnostic code we could add to Beta release builds to help pinpoint the cause of these crashes that spike intermittently?

This bug seems to be caused by "bad builds" because there are no suspicious commits in the beta builds that crash a lot. The crashes don't seem to be based on any external cause either because the crashes don't show up at all in any other releases during the same time period.

Comment 30

[Marco Castelluccio [:marco]]

The crash is gone again in 52.0b7.

Comment 31

[Julien Cristau [:jcristau]]

We still don't know what's causing these crashes that happen only on some builds, so wontfix for 52.

Comment 32

[Chris Peterson [:cpeterson]]

There are fewer than 50 crashes in Beta 54:

https://crash-stats.mozilla.com/search/?signature=~TypeAnalyzer%3A%3AinsertConversions&product=Firefox&version=54.0b&platform=Windows&date=>%3D2017-02-19T23%3A53%3A00.000Z&date=<2017-05-19T23%3A53%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Unless this crash spikes in Beta 54 or 55, I don't think it needs to block our win64 rollout (meta bug 1340936).

Comment 33

[Worcester12345]

Might just as well do a "WONTFIX" on firefox54 while you're at it. Best to concentrate on 55 and 56 at this point.

Comment 34

[Sylvestre Ledru [:Sylvestre]]

Closing because no crash reported since 12 weeks.