1279590 - Crash in jit | CORRUPT_CODE

Status: RESOLVED INCOMPLETE

(Core :: JavaScript Engine: JIT, defect, P3)

Description

[Marcia Knous [:marcia]]

[Tracking Requested - why for this release]: Right now #20 on 47 - probably should keep an eye on it.


This bug was filed from the Socorro interface and is 
report bp-44b87d4f-4be8-47db-bc7c-f52b32160609.
=============================================================

Seen while looking at 47 crashes. Right now sits at #20 with no bug associated with it. Link to crashes: http://bit.ly/1UHxYox

Highest percentage of crashes are Win 7/Win 10. This crash appears to have be present in 47.0b99 and then carried on into the release. 

The comments tab does have a few comments in English - Some mention upgrading and one specifically mentions "this is the second time in 10 minutes that firefox shutdown and show me the crash report "

Comment 1

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Jan, Naveed, I thought I'd start with you guys. Could you please take a quick look at some of the crash reports to see if we can root cause this? I am planning to do a dot release the week of 6/20 and if we have a safe fix by then, we can consider taking this one as a ride-along. Thanks!

Comment 2

[Jan de Mooij [:jandem]]

(In reply to Ritu Kothari (:ritu) from comment #1)
> Hi Jan, Naveed, I thought I'd start with you guys. Could you please take a
> quick look at some of the crash reports to see if we can root cause this? I
> am planning to do a dot release the week of 6/20 and if we have a safe fix
> by then, we can consider taking this one as a ride-along. Thanks!

Sorry for the delay. A portion of these are the mysterious AMD crashes, I filed bug 1281759 to work around those.

The remaining ones are likely random memory corruption for the most part. Let's start with bug 1281759 and see how much that helps.

Comment 3

[Gerry Chang [:gchang]]

Track this as the crash happens in 48/49/50.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'jit | CORRUPT_CODE':
 - esr (version 45): 72 crashes from 2016-04-07.

Crash volume on the last weeks:
             Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - nightly          5          2          4          3          4          2          0
 - aurora          18         24         18         23         22         21          3
 - beta           177         87         63         91         87         67         23
 - release       2065       2103       2147       2221       2304       2047        779
 - esr             13         14         12         10          7          5          5

Affected platform: Windows

Comment 5

[Gerry Chang [:gchang]]

Hi Jan,
The patch of bug 1281759 has been in m-c for a while and I can still see some crashes in 50/51, do you have any updates for this?

Comment 6

[Jan de Mooij [:jandem]]

(In reply to Gerry Chang [:gchang] from comment #5)
> Hi Jan,
> The patch of bug 1281759 has been in m-c for a while and I can still see
> some crashes in 50/51, do you have any updates for this?

Hi Gerry, the patch for bug 1281759 was never meant to fix all these crashes. A lot of them are caused by memory corruption (see bug 1034706 comment 44). This signature just showed up because we categorize JIT crashes now.

I also don't think it's useful to track this for Firefox 49/50, without anything actionable.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'jit | CORRUPT_CODE':
 - nightly (version 51): 12 crashes from 2016-08-01.
 - aurora  (version 50): 40 crashes from 2016-08-01.
 - beta    (version 49): 1964 crashes from 2016-08-02.
 - release (version 48): 235 crashes from 2016-07-25.
 - esr     (version 45): 261 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       6       2       1
 - aurora       14      11       7
 - beta        661     690     211
 - release      71      73      39
 - esr          50      56      39

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #223      #191
 - aurora  #400      #59
 - beta    #17       #12
 - release #260      #171
 - esr     #174

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

From comment 6, marking this wontfix for 49 since it seems unactionable. It is a pretty high volume crash on beta 49, though.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'jit | CORRUPT_CODE':
 - nightly (version 52): 11 crashes from 2016-09-19.
 - aurora  (version 51): 13 crashes from 2016-09-19.
 - beta    (version 50): 743 crashes from 2016-09-20.
 - release (version 49): 2394 crashes from 2016-09-05.
 - esr     (version 45): 441 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly       5       6
 - aurora       11       2
 - beta        585     158
 - release    1841     552
 - esr          16      27

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #214      #846
 - aurora  #253      #164
 - beta    #25       #19
 - release #33       #12
 - esr     #536

Comment 10

[Hannes Verschore [:h4writer]]

Marking P3 since we should move this forward. Per comment 2 the current way to drive this is to get bug 1281759 fixed. If that works we can come back trying to fix the left-overs.

Comment 11

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Hannes, bug 1281759 was fixed in 49 and 50 but this signature is still a really high volume crash on 49.0.2 and beta50. What do you suggest as next steps?

Comment 12

[Hannes Verschore [:h4writer]]

(In reply to Ritu Kothari (:ritu) from comment #11)
> Hi Hannes, bug 1281759 was fixed in 49 and 50 but this signature is still a
> really high volume crash on 49.0.2 and beta50. What do you suggest as next
> steps?

We identified two issues
1) Specific Amd crashes
2) Memory corruption

For (1) we talked to AMD about various fixes. We tried one in bug 1281759, but it didn't fix the crashes. I called out for the specific hardware and we have it since today and are trying to reproduce it locally. If we can we might find a work-around faster.

For (2) there is not really a plan forward. This is just random memory corruption. That can happen in any code. Sadly we don't have a good way to categorize those ?yet?

Comment 13

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Hannes. I still have one beta build to push this week and RC week starting next week. Given the high volume here if there is a safe fix, we should consider uplifting to 50.

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Updating the status flags as it seems that we won't be able to fix that soon but happy to take a patch

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'jit | CORRUPT_CODE':
 - nightly (version 54): 12 crashes from 2017-01-23.
 - aurora  (version 53): 17 crashes from 2017-01-23.
 - beta    (version 52): 984 crashes from 2017-01-23.
 - release (version 51): 3308 crashes from 2017-01-16.
 - esr     (version 45): 866 crashes from 2016-08-10.

Crash volume on the last weeks (Week N is from 02-06 to 02-12):
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       8       4
 - aurora       12       1
 - beta        578     217
 - release    2085     555       1
 - esr          23      38      44      24      35      26      37

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content   Plugin
 - nightly #249      #154
 - aurora  #242      #60
 - beta    #24       #23
 - release #38       #10
 - esr     #318

Comment 16

[Ted Campbell [:tcampbell]]

Signature is no longer valid. Closing bug in favour of Bug 858032.