Closed Bug 1276558 Opened 4 years ago Closed Last year

Crash in `anonymous namespace''::TypeAnalyzer::insertConversions

Categories

(Core :: JavaScript Engine: JIT, defect, P3, critical)

Unspecified
Windows
defect

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox47 --- wontfix
firefox48 --- wontfix
firefox49 --- wontfix
firefox-esr45 --- wontfix
firefox50 - wontfix
firefox51 - wontfix
firefox52 + wontfix
firefox53 --- wontfix
firefox54 --- affected
firefox55 --- affected

People

(Reporter: njn, Unassigned)

References

()

Details

(Keywords: crash)

Crash Data

This bug was filed from the Socorro interface and is 
report bp-e511ea20-bdd3-428e-8df0-1b6d42160529.
=============================================================

31 crashes with this signature in the past 7 days. Crash address is always 0x0. I looked at a minidump and the crashing instruction was this:

> call qword ptr [r9]

and r9 was 0x0. I'm not sure what this means -- is |iter| null? is adjustInputs somehow null (bad vtable)?

Jan, any ideas?
Flags: needinfo?(jdemooij)
(In reply to Nicholas Nethercote [:njn] from comment #0)
> Crash address is always
> 0x0. I looked at a minidump and the crashing instruction was this:
> 
> > call qword ptr [r9]
> 
> and r9 was 0x0. I'm not sure what this means -- is |iter| null? is
> adjustInputs somehow null (bad vtable)?
> 
> Jan, any ideas?

I looked at bp-9565cba8-aeed-4392-9e78-4c9d82160525.

We're crashing in TypeAnalyzer::adjustInputs. It's inlined so the debugger gets confused. The (virtual) policy->adjustInputs call here crashes:

    if (policy && !policy->adjustInputs(alloc(), ins))
        return false;

The vtable pointer is in rax, that's an address inside libxul. If I look at the memory view, I see many pages filled with zeroes around this address. Not sure what's going on there - it could be a bogus vtable but the address is not entirely bogus..

The list of crashes here shows that the uptime is extremely low for these crashes on Nightly (1, 1, 1, 2, 3, 4 seconds):

https://crash-stats.mozilla.com/signature/?date=%3C2016-05-31T12%3A16%3A34.360318&version=49.0a1&signature=%60anonymous%20namespace%27%27%3A%3ATypeAnalyzer%3A%3AinsertConversions&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=uptime&page=1#reports
Flags: needinfo?(jdemooij)
Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly(version 50):55 crashes from 2016-06-06.
 - aurora (version 49):60 crashes from 2016-06-07.
 - beta   (version 48):62 crashes from 2016-06-06.
 - release(version 47):41 crashes from 2016-05-31.
 - esr    (version 45):5 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly      18       4       1       6       3      11       4
 - aurora       36       5       1       1       3       8       3
 - beta          9       5      10       7       8      10      12
 - release       4       9       4       3       4       4       9
 - esr           0       0       3       1       0       0       0

Affected platform: Windows
Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly (version 51): 4 crashes from 2016-08-01.
 - aurora  (version 50): 4 crashes from 2016-08-01.
 - beta    (version 49): 25 crashes from 2016-08-02.
 - release (version 48): 21 crashes from 2016-07-25.
 - esr     (version 45): 8 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       2       1       0
 - aurora        1       1       0
 - beta         10      13       1
 - release       6       5       5
 - esr           0       0       1

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #298
 - aurora  #399
 - beta    #3016     #1919
 - release #2422
 - esr     #2187
Duplicate of this bug: 1276554
this signature seems to spike in early data for the 49.0b10 release, where it's also a startup crash.
Keywords: topcrash
Not great but also not a huge crash volume. I don't think this should be a release blocker for 49.
[Tracking Requested - why for this release]:
this is currently the #3 startup crasher for 49.0.1 (0.36% of all browser crashes, 2% of startup crashes) - could we look into it more trying to get it fixed in upcoming releases?
OS: Windows 8 → Windows
Hi Jon, Terrence, given the severity (see comment 7), would you be able to help investigate?
Flags: needinfo?(terrence.d.cole)
Flags: needinfo?(jcoppeard)
Do we have reason at all to expect that this is GC related? Unless we have STR, we're not going to be able to tell you anything more than what Jan already discovered in comment 1.
Flags: needinfo?(terrence.d.cole)
This crash is mostly non-existent, up through 49.0b10, judging by comment 5 and the release management bot posts before that. What landed in that version of beta? Maybe something there is to blame.
(100.0% in signature vs 05.22% overall) cpu_arch = amd64
(100.0% in signature vs 10.86% overall) address = 0x0
(100.0% in signature vs 34.81% overall) reason = EXCEPTION_ACCESS_VIOLATION_READ
(65.85% in signature vs 25.56% overall) platform_pretty_version = Windows 10
(100.0% in signature vs 67.04% overall) dom_ipc_enabled = null
(43.53% in signature vs 12.88% overall) platform_version = 10.0.14393
(19.87% in signature vs 00.54% overall) cpu_info = family 6 model 58 stepping 9 | 4

"dom_ipc_enabled = null" could be spurious, since the crash is happening
really early at startup.
Cancelling my needinfo since this does not appear to be GC related.
Flags: needinfo?(jcoppeard)
I can't imagine anything that landed between b9 and b10 made a difference here. The fact that it's AMD-only is interesting.
Priority: -- → P3
(In reply to Till Schneidereit [:till] from comment #14)
> I can't imagine anything that landed between b9 and b10 made a difference
> here. The fact that it's AMD-only is interesting.

cpu_arch=amd64 actually means the target architecture of the Firefox
build, so this is Firefox 64-bit only.
(In reply to Marco Castelluccio [:marco] from comment #15)
> cpu_arch=amd64 actually means the target architecture of the Firefox
> build, so this is Firefox 64-bit only.

Ugh, of course. The fact that it's 64bit-only is interesting, too, though :)
I only see 1 instance of this crash on Beta50 in the past 7 days, don't feel the need to track it and updating the status to wontfix for 50.
Crash volume for signature '`anonymous namespace''::TypeAnalyzer::insertConversions':
 - nightly (version 52): 9 crashes from 2016-09-19.
 - aurora  (version 51): 3 crashes from 2016-09-19.
 - beta    (version 50): 6 crashes from 2016-09-20.
 - release (version 49): 4208 crashes from 2016-09-05.
 - esr     (version 45): 10 crashes from 2016-07-25.

Crash volume on the last weeks (Week N is from 10-17 to 10-23):
            W. N-1  W. N-2  W. N-3  W. N-4
 - nightly       4       4       0       0
 - aurora        0       0       2       0
 - beta          2       0       4       0
 - release    1364    1199    1077     161
 - esr           3       1       0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #260      #203
 - aurora  #1321
 - beta    #5238
 - release #33       #2346
 - esr     #2575
This is the top crash for 64-bit Release 49.
Blocks: support-win64
No longer blocks: tracking_win64
@ Liz: were there any changes in 49.0.1 besides the DLL shim for Websense bug 1304783? Was that issue subsequently fixed in 49.0.2?

https://www.mozilla.org/en-US/firefox/49.0.1/releasenotes/

This signature spiked in Release 49.0.1 but only for 64-bit Firefox. Crash reports per 49.0.x dot-release:

Firefox  32-bit  64-bit
=======  ======  ======
 49.0.0       0       1
 49.0.1      12    4963 !!!
 49.0.2       9      56
Depends on: 1304783
Flags: needinfo?(lhenry)
Keywords: topcrash
Chris: we only shipped 49.0 to a small segment of the user population (the first day of the release); then we throttled updates and put out 49.0.1. Yes, as far as I can tell, the websense DLL shim was the only change between 49.0 and 49.0.1.  I wouldn't expect to see this crash show up significantly in 49.0. 

And here's the changes for 49.0.2: https://www.mozilla.org/en-US/firefox/49.0.2/releasenotes/ 
Do you think something we added to 49.0.2 fixed this? I still see some crashes for 49.0.2 (and in 50/51)
Flags: needinfo?(lhenry)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> Chris: we only shipped 49.0 to a small segment of the user population (the
> first day of the release); then we throttled updates and put out 49.0.1.
> Yes, as far as I can tell, the websense DLL shim was the only change between
> 49.0 and 49.0.1.  I wouldn't expect to see this crash show up significantly
> in 49.0. 

I see. So this is probably not a regression in 49.0.1. Comment 5 says this crash first spiked in Beta 49.0b10.

> And here's the changes for 49.0.2:
> https://www.mozilla.org/en-US/firefox/49.0.2/releasenotes/ 
> Do you think something we added to 49.0.2 fixed this? I still see some
> crashes for 49.0.2 (and in 50/51)

None of the changes in 49.0.2 look like obvious fixes. Perhaps this crash is caused by some random 64-bit-only code-generation bug at compile time? I don't really know.
Track 51- as the volume of crash in 51 is very low.
this is spiking up again in win64 builds of firefox 52.0b6
(In reply to [:philipp] from comment #24)
> this is spiking up again in win64 builds of firefox 52.0b6

Does this look like a regression in 52.0b6 or just coincidental timing? Are these crashes also spiking in 52.0b5 or 51.0 during this same time period?
the spike is just in beta6 - so this bug may well be some random build-specific error that pops up every now & then like you suggested in comment #22
Here are the changes between 52.0b5 and 52.0b6. None of them stand out as likely suspects for causing 52.0b6's spike in JIT crashes.

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_52_0b5_RELEASE&tochange=FIREFOX_52_0b6_RELEASE
(In reply to Chris Peterson [:cpeterson] from comment #27)
> Here are the changes between 52.0b5 and 52.0b6. None of them stand out as
> likely suspects for causing 52.0b6's spike in JIT crashes.
> 
> https://hg.mozilla.org/releases/mozilla-beta/
> pushloghtml?fromchange=FIREFOX_52_0b5_RELEASE&tochange=FIREFOX_52_0b6_RELEASE

I also don't see anything particular for the jits.
Jan, based on your initial investigation in comment 1, is there any diagnostic code we could add to Beta release builds to help pinpoint the cause of these crashes that spike intermittently?

This bug seems to be caused by "bad builds" because there are no suspicious commits in the beta builds that crash a lot. The crashes don't seem to be based on any external cause either because the crashes don't show up at all in any other releases during the same time period.
Flags: needinfo?(jdemooij)
The crash is gone again in 52.0b7.
We still don't know what's causing these crashes that happen only on some builds, so wontfix for 52.
There are fewer than 50 crashes in Beta 54:

https://crash-stats.mozilla.com/search/?signature=~TypeAnalyzer%3A%3AinsertConversions&product=Firefox&version=54.0b&platform=Windows&date=>%3D2017-02-19T23%3A53%3A00.000Z&date=<2017-05-19T23%3A53%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Unless this crash spikes in Beta 54 or 55, I don't think it needs to block our win64 rollout (meta bug 1340936).
Flags: needinfo?(jdemooij)
Might just as well do a "WONTFIX" on firefox54 while you're at it. Best to concentrate on 55 and 56 at this point.
Closing because no crash reported since 12 weeks.
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.