1192988 - Crash in js::jit::ICStub::traceCode

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine: JIT, defect, P3)

Description

[Jim Mathies [:jimm]]

currently #14 top crash in nightly:

https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2015-08-10&signature=js%3A%3Ajit%3A%3AICStub%3A%3Atrace%28JSTracer*%29&version=Firefox%3A42.0a1#tab-sigsummary

report bp-465efa92-1586-4a9c-9d81-6842f2150809.
=============================================================

Comment 1

[Semtex]

I'm able to crash Nightly in easy way with this crash sig:
https://crash-stats.mozilla.com/report/index/bp-bf822daf-3b44-4bee-a2c7-f5ef82160322
https://crash-stats.mozilla.com/report/index/65c74020-0a8b-48b6-9b05-785432160322
https://crash-stats.mozilla.com/report/index/bp-bbe582ca-1c04-476b-acb3-59b362160322
Nightly and OS info in crash sig.

To reproduce You need IPB forum 4.0 version

1.Open existing topic
2.Use editor- choose [code] tag
3.In Code popup click with right mouse button
4.Nightly will crash with crash sig from above.


Since I'm admin on IPB 4.0 forum I can create test account for tests, but login and password need to be used by one person  working on this bug. I will not post this on open channel.

Comment 2

[alex_mayorga]

¡Hola!

Found this crash from https://support.mozilla.org/en-US/questions/1114853#answer-858611

846 crashes (194 startup) in the past week per https://crash-stats.mozilla.com/report/list?product=Firefox&signature=js%3A%3Ajit%3A%3AICStub%3A%3Atrace

¡Gracias!
Alex

Comment 3

[Jan de Mooij [:jandem]]

Hi Semtex, your crashes are probably a duplicate of bug 1258301. I just landed a fix for that.

Once that hits the Nightly channel, it'd be great if you could confirm it no longer crashes.

Comment 4

[Semtex]

(In reply to Jan de Mooij [:jandem] from comment #3)
> Hi Semtex, your crashes are probably a duplicate of bug 1258301. I just
> landed a fix for that.
> 
> Once that hits the Nightly channel, it'd be great if you could confirm it no
> longer crashes.

Feedback, I'm not able to crash anymore with crash sig from my first post, so Your fix seems to be working, now I'm able to crash with same STR to this crash SIG: https://crash-stats.mozilla.com/report/index/a3d93ec1-2958-4232-8d44-1b8fd2160325
lol

Comment 5

[Jan de Mooij [:jandem]]

(In reply to Semtex from comment #4)
> now I'm able to crash with same STR to this crash SIG:
> https://crash-stats.mozilla.com/report/index/a3d93ec1-2958-4232-8d44-
> 1b8fd2160325
> lol

Thanks for your feedback. Actually that build does not include my fix yet so this crash is likely very similar to your other signature :)

Comment 6

[Semtex]

Maybe Your fix, when landed will fix both crashes, hope so... ;)

Comment 7

[Jan de Mooij [:jandem]]

Semtex, does the latest Nightly still crash? :)

Comment 8

[Semtex]

(In reply to Jan de Mooij [:jandem] from comment #7)
> Semtex, does the latest Nightly still crash? :)

Hi, I'm not able to crash Nightly, so looks like fixed. Thanks.
If I will see some new crash i will inform about that ASAP.

Comment 9

[Jan de Mooij [:jandem]]

(In reply to Semtex from comment #8)
> Hi, I'm not able to crash Nightly, so looks like fixed. Thanks.
> If I will see some new crash i will inform about that ASAP.

Great, thank you.

Let's keep this bug open for comment 0.

Comment 10

[Calixte Denizet (:calixte)]

Crash volume for signature 'js::jit::ICStub::trace':
  - nightly (50): 5
  - aurora (49): 10
  - esr (45): 107

Affected platforms: Windows, Mac OS X, Linux

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'js::jit::ICStub::trace':
 - nightly (version 51): 2 crashes from 2016-08-01.
 - aurora  (version 50): 2 crashes from 2016-08-01.
 - beta    (version 49): 33 crashes from 2016-08-02.
 - release (version 48): 84 crashes from 2016-07-25.
 - esr     (version 45): 181 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       1       0       1
 - aurora        2       0       0
 - beta         11      12       5
 - release      38      14      11
 - esr          18      11      21

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly
 - aurora  #599
 - beta    #2012     #801
 - release #549
 - esr     #354

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'js::jit::ICStub::trace':
 - nightly (version 52): 4 crashes from 2016-09-19.
 - aurora  (version 51): 2 crashes from 2016-09-19.
 - beta    (version 50): 8 crashes from 2016-09-20.
 - release (version 49): 1362 crashes from 2016-09-05.
 - esr     (version 45): 273 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly       2       2
 - aurora        2       0
 - beta          6       2
 - release    1065     297
 - esr          12      26

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #445
 - aurora  #654
 - beta    #5063     #522
 - release #69       #22
 - esr     #592

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'js::jit::ICStub::trace':
 - nightly (version 53): 41 crashes from 2016-11-14.
 - aurora  (version 52): 45 crashes from 2016-11-14.
 - beta    (version 51): 913 crashes from 2016-11-14.
 - release (version 50): 3998 crashes from 2016-11-01.
 - esr     (version 45): 593 crashes from 2016-07-06.

Crash volume on the last weeks (Week N is from 01-02 to 01-08):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       8       8       8       9       7       0       0
 - aurora       13      11       8       5       1       1       0
 - beta        303     291     194      13      10       7       7
 - release    1323    1338     673      76      81      70      16
 - esr          21      35      32      37      29      39      22

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly #643      #104
 - aurora  #630      #133
 - beta    #159      #60
 - release #161      #68
 - esr     #408

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'js::jit::ICStub::trace':
 - nightly (version 54): 13 crashes from 2017-01-23.
 - aurora  (version 53): 5 crashes from 2017-01-23.
 - beta    (version 52): 204 crashes from 2017-01-23.
 - release (version 51): 728 crashes from 2017-01-16.
 - esr     (version 45): 681 crashes from 2016-08-03.

Crash volume on the last weeks (Week N is from 01-30 to 02-05):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       6
 - aurora        2
 - beta        105
 - release     345       0
 - esr          49      45      41      32      21      35      32

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly           #40
 - aurora  #221      #76
 - beta    #70       #29
 - release #63       #21
 - esr     #372

Comment 15

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

Wild pointers, some of them clear UAFs, in GC.

Comment 17

[Naveed Ihsanullah [:naveed]]

Jon is this still s-s after Jan's patches?

Comment 18

[Jon Coppeard (:jonco)]

It looks like this crash is still happening so probably safest to leave this s-s.

Comment 19

[Jon Coppeard (:jonco)]

Unassigning myself as I'm not actively working on this.

Comment 20

[Daniel Veditz [:dveditz]]

Steve: please find an appropriate assignee for this bug (or keep it for yourself if you don't have anyone at the moment). We need an assigned responsible person for severe security bugs.

Comment 21

[Sylvestre Ledru [:Sylvestre]]

Andrew, could you please help with this bug? This is happening for a while and the volume isn't negligible.
Thanks

Comment 22

[Andrew Overholt [:overholt]]

Steve and I will try to get some fresh eyes on this to make progress.

Comment 23

[Steven DeTar [:sdetar]]

sstangl: Would you be willing to look at this bug?

Comment 24

[Steve Fink [:sfink] [:s:]]

Skimming stacks, I'm seeing:

JSScript::traceChildren -> BaselineScript::trace -> ICStub::trace
(or through BaselineICEntry::trace)

JSScript::markChildren -> TraceIonScripts -> BaselineScript::Trace -> ... -> ICStub::trace

and then crashing on a wide variety of different addresses.

Which makes me wonder -- are we tracing dead scripts, jitscripts, or stubs? I wonder if a JIT person could look at the minidumps and tell where we're dealing with real objects and when we start dealing with garbage memory. (I haven't decompiled to see what instructions are crashing.)

Comment 25

[Ted Campbell [:tcampbell]]

(Initial instinct is corruption, but whodunit?)

Comment 26

[Ted Campbell [:tcampbell]]

Absorb Bug 1260721 since that seems to be a inlining-dependent version of this. It is probably worth noting the proportion of these trace crashes that are in traceCode vs the aggregate.

Comment 27

[Ted Campbell [:tcampbell]]

Almost all the ICStub::trace crash volume is absorbed from ICStub::traceCode depending on inlining. The traceCode is the first line of trace anyways.

This signature is a mess similar to Bug 858032 and at least a fraction of it is hardware failure or single-bit errors. The 70% Windows 7 bias is certainly curious. I'll look at minidumps.

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

This is still affecting 62/63 on release at a fairly high rate.

Comment 29

[Ted Campbell [:tcampbell]]

I'll ask iain to look into mitigation described in Bug 1462104.

Comment 30

[alex_mayorga]

¡Hola!

Updated the flags per the "Crash Data".

¡Gracias!
Alex

Comment 31

[Iain Ireland [:iain]]

We've collected about a month of data from the work in bug 1462104. Of the 34 crashes in ICEntry::trace, 27 either triggered the new assertion, or crashed while trying to evaluate the new assertion. This implies that ICEntry::trace was called with a bad |this| pointer.

Most of the remaining crashes occur when we try to dereference the jitcode_ pointer. Not all of the crash dumps contain useful data, but both cases that tcampbell and I managed to investigate were clear bitflips: the faulting pointer differed in only one bit from a valid pointer to executable memory.

There doesn't appear to be any signal in this noise. This signature is just what happens when you have a bunch of pointers that all get dereferenced by the same few lines of code.

Closing as WORKSFORME.